An Empirical Study of the Non-determinism of ChatGPT in Code Generation

TL;DR

This study systematically quantifies the non-determinism of ChatGPT in Python code generation across CodeContests, APPS, and HumanEval by analyzing five predictions per task along semantic, syntactic, and structural dimensions. It shows high non-determinism under default settings, with temperature adjustments not guaranteeing determinism, and finds that prompt design and task features influence variability. The work also compares GPT-4 and GPT-3.5, and discusses the implications for reproducibility and experimental validity in LLM-based software engineering research, recommending practices such as reporting variance and using diverse benchmarks. Overall, the findings call for methodological safeguards to ensure reliable scientific conclusions and practical robustness when deploying LLMs for code tasks.

Abstract

There has been a recent explosion of research on Large Language Models (LLMs) for software engineering tasks, in particular code generation. However, results from LLMs can be highly unstable; nondeterministically returning very different codes for the same prompt. Non-determinism is a potential menace to scientific conclusion validity. When non-determinism is high, scientific conclusions simply cannot be relied upon unless researchers change their behaviour to control for it in their empirical analyses. This paper conducts an empirical study to demonstrate that non-determinism is, indeed, high, thereby underlining the need for this behavioural change. We choose to study ChatGPT because it is already highly prevalent in the code generation research literature. We report results from a study of 829 code generation problems from three code generation benchmarks (i.e., CodeContests, APPS, and HumanEval). Our results reveal high degrees of non-determinism: the ratio of coding tasks with zero equal test output across different requests is 75.76%, 51.00%, and 47.56% for CodeContests, APPS, and HumanEval, respectively. In addition, we find that setting the temperature to 0 does not guarantee determinism in code generation, although it indeed brings less non-determinism than the default configuration (temperature=1). These results confirm that there is, currently, a significant threat to scientific conclusion validity. In order to put LLM-based research on firmer scientific foundations, researchers need to take into account non-determinism in drawing their conclusions.

Figures (6)

• Figure 1: Overview of the experimental procedure.
• Figure 2: RQ1.1: Distribution of semantic similarity in terms of test pass rate.
• Figure 3: RQ1.1: Distribution of semantic similarity in terms of test output equivalence rate (OER and OER (no ex.)).
• Figure 4: RQ1.2: Distribution of syntactic similarity (LCS & LED). Lower LCS and higher LED indicate less syntactic similarity.
• Figure 5: RQ1.3: Structural Similarity (United_Diff & Tree_Diff).