Can OpenSource beat ChatGPT? -- A Comparative Study of Large Language Models for Text-to-Code Generation
Luis Mayer, Christian Heumann, Matthias Aßenmacher
TL;DR
This study benchmarks five LLMs (Bard, BingChat, ChatGPT, Llama2, Code Llama) on text-to-code generation using LeetCode Python tasks to quantify correctness, runtime, and memory usage. Using a standardized prompt structure and LeetCode testing, the authors reveal substantial differences in performance, with ChatGPT consistently achieving the highest correctness and competitive runtime/memory characteristics, while open-source models lag substantially in correctness. The work also analyzes error types and how prompt design and indentation issues influence outcomes, highlighting practical considerations for deploying LLMs in code-generation tasks. The findings suggest that, as of the study, GPT-based models remain superior for Python text-to-code, though prompt engineering and post-processing strategies can meaningfully affect results. Overall, the paper provides a snapshot of model capabilities, informs future open-source improvements, and emphasizes the importance of prompt design and output formatting in automated code generation.
Abstract
In recent years, large language models (LLMs) have emerged as powerful tools with potential applications in various fields, including software engineering. Within the scope of this research, we evaluate five different state-of-the-art LLMs - Bard, BingChat, ChatGPT, Llama2, and Code Llama - concerning their capabilities for text-to-code generation. In an empirical study, we feed prompts with textual descriptions of coding problems sourced from the programming website LeetCode to the models with the task of creating solutions in Python. Subsequently, the quality of the generated outputs is assessed using the testing functionalities of LeetCode. The results indicate large differences in performance between the investigated models. ChatGPT can handle these typical programming challenges by far the most effectively, surpassing even code-specialized models like Code Llama. To gain further insights, we measure the runtime as well as the memory usage of the generated outputs and compared them to the other code submissions on Leetcode. A detailed error analysis, encompassing a comparison of the differences concerning correct indentation and form of the generated code as well as an assignment of the incorrectly solved tasks to certain error categories allows us to obtain a more nuanced picture of the results and potential for improvement. The results also show a clear pattern of increasingly incorrect produced code when the models are facing a lot of context in the form of longer prompts.