Assessing the Promise and Pitfalls of ChatGPT for Automated Code Generation

TL;DR

This work provides a rigorous, data-driven assessment of ChatGPT's code-generation prowess relative to human programmers using a bespoke dataset of 131 prompts across five domains. It leverages 14 code-quality metrics and manual evaluation to characterize correctness, understandability, and security, while also introducing machine-learning-based methods to distinguish AI-generated from human-written code. Key findings show ChatGPT excels in data-analysis tasks and produces maintainable, well-handled code but struggles with visual/graphical tasks and may rely on outdated package knowledge. The study offers a robust benchmark, a publicly available dataset, and ML-based detection methods to guide future AI-assisted programming tools and education research.

Abstract

This paper presents a comprehensive evaluation of the code generation capabilities of ChatGPT, a prominent large language model, compared to human programmers. A novel dataset of 131 code-generation prompts across 5 categories was curated to enable robust analysis. Code solutions were generated by both ChatGPT and humans for all prompts, resulting in 262 code samples. A meticulous manual assessment methodology prioritized evaluating correctness, comprehensibility, and security using 14 established code quality metrics. The key findings reveal ChatGPT's strengths in crafting concise, efficient code with advanced constructs, showcasing strengths in data analysis tasks (93.1% accuracy) but limitations in visual-graphical challenges. Comparative analysis with human code highlights ChatGPT's inclination towards modular design and superior error handling. Additionally, machine learning models effectively distinguished ChatGPT from human code with up to 88% accuracy, suggesting detectable coding style disparities. By providing profound insights into ChatGPT's code generation capabilities and limitations through quantitative metrics and qualitative analysis, this study makes valuable contributions toward advancing AI-based programming assistants. The curated dataset and methodology offer a robust foundation for future research in this nascent domain. All data and codes are available on https://github.com/DSAatUSU/ChatGPT-promises-and-pitfalls.

Figures (8)

• Figure 1: Categories and subcategories for which data and prompts were collected. The numbers show the total number of prompts collected for that category or subcategory.
• Figure 2: Code generation script for automatically processing each prompt and exporting code into specified directories.
• Figure 3: Codes and outputs for the presented case studies
• Figure 4: This box and whisker plot illustrates key values of analytical metrics: Median (line within the box), Mean ('x' symbol), Minimum (lower whisker), Maximum (upper whisker), Lower Quartile (bottom of the box), and Upper Quartile (top of the box) for each metric
• Figure 5: This radar plot displays the mean for each metric across categories, providing a comprehensive view of ChatGPT and human coding styles across categories