The Impact of the Prompt Language on Code Generation with LLMs: An Experimental Study

This study investigates the effect of using different languages in the prompt input for code generation with large language models (LLMs). Python code is generated from German and English prompt inputs using a zero-shot prompting strategy, targeting a comparison of the model performance between the languages and difficulties. Therefore, a bilingual benchmark dataset containing 16 algorithmic tasks was developed, covering domains such as linear algebra, number theory, probability theory, and cryptography. The tasks were semantically aligned and categorized into three levels of difficulty. Six large language models were evaluated across the tasks. Standardized prompts were designed using prompt engineering principles, assigning the models the role of an expert in the topic of Python programming. Score metrics were implemented to assess the individual performance of the models and statistical analyses were conducted using Pearson’s chi-squared and Fisher exact test to evaluate significant associations across language prompts and task difficulty. Each model completed 16 tasks in German and 16 in English, with two independent runs per task, resulting in 64 runs per model. This adds up to 192 test runs per language and 384 runs in total. Results show that most models perform better using English language prompts, with GPT-4o outperforming its competitors, reaching the highest scores in both languages and solving all tasks with the maximum score with English prompting. Association between the models across the difficulties showed significant differences using English prompting, while for German only the easiest difficulty showed significant differences. Association between the language performances only showed significant differences in medium difficulty with Pearson’s chi-squared test ( $$p=0.05$$ p = 0.05 ) and Fisher exact test ( $$p=0.029$$ p = 0.029 ).