Evaluation of the Programming Skills of Large Language Models
Luc Bryan Heitz, Joun Chamas, Christopher Scherb
TL;DR
The paper evaluates the programming-code quality produced by free versions of OpenAI's ChatGPT and Google's Gemini AI using HumanEval and ClassEval datasets, complemented by a hands-on Java project. It employs compilation, semantic, and practical implementation tests to quantify functional correctness, maintainability indicators like code smells, and real-world applicability. Results show ChatGPT achieving higher functional pass rates across datasets, but both models produce semantic errors and code smells that necessitate thorough review and testing. The study emphasizes rigorous QA when integrating AI-generated code into software projects and outlines future work on premium models, real-world industry use, and AI-driven unit-test generation.
Abstract
The advent of Large Language Models (LLM) has revolutionized the efficiency and speed with which tasks are completed, marking a significant leap in productivity through technological innovation. As these chatbots tackle increasingly complex tasks, the challenge of assessing the quality of their outputs has become paramount. This paper critically examines the output quality of two leading LLMs, OpenAI's ChatGPT and Google's Gemini AI, by comparing the quality of programming code generated in both their free versions. Through the lens of a real-world example coupled with a systematic dataset, we investigate the code quality produced by these LLMs. Given their notable proficiency in code generation, this aspect of chatbot capability presents a particularly compelling area for analysis. Furthermore, the complexity of programming code often escalates to levels where its verification becomes a formidable task, underscoring the importance of our study. This research aims to shed light on the efficacy and reliability of LLMs in generating high-quality programming code, an endeavor that has significant implications for the field of software development and beyond.