Not the Silver Bullet: LLM-enhanced Programming Error Messages are Ineffective in Practice

TL;DR

This study investigates whether GPT-4 enhanced programming error messages (PEMs) improve novice debugging in real practice. In a within-subjects design with $n=106$ participants who fix $6$ buggy C programs under three PEM styles (GCC, handwritten, GPT-4), GPT-4 explanations rarely sped up debugging, succeeding on only one task, while expert handwritten explanations outperform both GPT-4 and GCC across multiple tasks. Despite the productivity gap, learners report higher satisfaction with GPT-4 and handwritten PEMs than with terse GCC messages, suggesting usability advantages that do not translate into faster task completion. The findings imply that LLM-based PEMs are not a universal solution for debugging, highlight the value of carefully designed handwritten diagnostics, and motivate further work to understand factors that make error messages usable for novices.

Abstract

The sudden emergence of large language models (LLMs) such as ChatGPT has had a disruptive impact throughout the computing education community. LLMs have been shown to excel at producing correct code to CS1 and CS2 problems, and can even act as friendly assistants to students learning how to code. Recent work shows that LLMs demonstrate unequivocally superior results in being able to explain and resolve compiler error messages -- for decades, one of the most frustrating parts of learning how to code. However, LLM-generated error message explanations have only been assessed by expert programmers in artificial conditions. This work sought to understand how novice programmers resolve programming error messages (PEMs) in a more realistic scenario. We ran a within-subjects study with $n$ = 106 participants in which students were tasked to fix six buggy C programs. For each program, participants were randomly assigned to fix the problem using either a stock compiler error message, an expert-handwritten error message, or an error message explanation generated by GPT-4. Despite promising evidence on synthetic benchmarks, we found that GPT-4 generated error messages outperformed conventional compiler error messages in only 1 of the 6 tasks, measured by students' time-to-fix each problem. Handwritten explanations still outperform LLM and conventional error messages, both on objective and subjective measures.

Figures (4)

• Figure 1: Examples of the three error message styles (i.e., study conditions) as they would appear to participants.
• Figure 2: Screenshot of the web-based IDE, showing the "instead of " task under the handwritten error message condition.
• Figure 3: Density plots of the log-transformed time-to-fix, by task. The black vertical line denotes the median.
• Figure 4: Proportion of participants' Likert responses. From top-to-bottom, the questions were "The message was easy to understand", "The message helped me understand what was wrong with the code", "The message helped me fix the code", and "The message was useful for fixing the problem".