Examination of Code generated by Large Language Models
Robin Beer, Alexander Feix, Tim Guttzeit, Tamara Muras, Vincent Müller, Maurice Rauscher, Florian Schäffler, Welf Löwe
TL;DR
This study systematically evaluates large language models’ ability to generate correct and high-quality code and unit tests across Java and Python for 12 algorithms, using ChatGPT and GitHub Copilot. It introduces a longitudinal evaluation framework with diverse prompts, manual corrections, and multi-faceted metrics (code correctness, code quality, test coverage, and test modification rate), allowing comparisons over time and against a prior baseline. Key findings show ChatGPT generally achieves higher code correctness, while Copilot yields strong code quality; Java outputs outperform Python in most metrics, and test-case generation remains more challenging than source-code generation. The work provides a reproducible benchmarking setup and data to guide future improvements in LLM-assisted software development and prompts.
Abstract
Large language models (LLMs), such as ChatGPT and Copilot, are transforming software development by automating code generation and, arguably, enable rapid prototyping, support education, and boost productivity. Therefore, correctness and quality of the generated code should be on par with manually written code. To assess the current state of LLMs in generating correct code of high quality, we conducted controlled experiments with ChatGPT and Copilot: we let the LLMs generate simple algorithms in Java and Python along with the corresponding unit tests and assessed the correctness and the quality (coverage) of the generated (test) codes. We observed significant differences between the LLMs, between the languages, between algorithm and test codes, and over time. The present paper reports these results together with the experimental methods allowing repeated and comparable assessments for more algorithms, languages, and LLMs over time.