How Accurately Do Large Language Models Understand Code?

TL;DR

This paper tackles the problem of how accurately large language models understand code, proposing a scalable, mutation-testing-inspired framework to quantify code comprehension via fault localization. It automatically generates thousands of debugging tasks by injecting faults and applying semantic-preserving mutations to real-world Python and Java programs, and evaluates nine LLMs in two phases to assess both basic fault detection and deeper semantic understanding. The findings reveal that LLMs exhibit shallow code understanding, with substantial performance degradation under semantic-preserving mutations and a strong dependence on non-functional cues, particularly variable names and comments. The work highlights significant implications for the use of LLMs in post-development tasks and suggests directions such as code-aware representations (eg, CFGs, Code Property Graphs) to improve robustness and cross-language reasoning in software engineering contexts.

Abstract

Large Language Models (LLMs) are increasingly used in post-development tasks such as code repair and testing. A key factor in these tasks' success is the model's deep understanding of code. However, the extent to which LLMs truly understand code remains largely unevaluated. Quantifying code comprehension is challenging due to its abstract nature and the lack of a standardized metric. Previously, this was assessed through developer surveys, which are not feasible for evaluating LLMs. Existing LLM benchmarks focus primarily on code generation, fundamentally different from code comprehension. Additionally, fixed benchmarks quickly become obsolete as they become part of the training data. This paper presents the first large-scale empirical investigation into LLMs' ability to understand code. Inspired by mutation testing, we use an LLM's fault-finding ability as a proxy for its deep code understanding. This approach is based on the insight that a model capable of identifying subtle functional discrepancies must understand the code well. We inject faults in real-world programs and ask the LLM to localize them, ensuring the specifications suffice for fault localization. Next, we apply semantic-preserving code mutations (SPMs) to the faulty programs and test whether the LLMs still locate the faults, verifying their confidence in code understanding. We evaluate nine popular LLMs on 600,010 debugging tasks from 670 Java and 637 Python programs. We find that LLMs lose the ability to debug the same bug in 78% of faulty programs when SPMs are applied, indicating a shallow understanding of code and reliance on features irrelevant to semantics. We also find that LLMs understand code earlier in the program better than later. This suggests that LLMs' code comprehension remains tied to lexical and syntactic features due to tokenization designed for natural languages, which overlooks code semantics.

Figures (12)

• Figure 1: N-Queen program, $P_{F}$ , with an injected fault.
• Figure 2: Output of OpenAI's GPT 4o when asked about the N-Queen problem spec (first), asked for faulty line in the faulty code ($P_F$) (second), and asked for faulty line in the faulty code with semantic-preserving mutations ($P_{F+SPM}$) (third) .
• Figure 3: N-Queen program, $P_{F+SPM}$, with an injected bug in Line 11 and Semantic Preserving Mutations.
• Figure 4: Phase I involves injecting faults into seed programs, testing LLMs on debugging tasks, and filtering out underspecified programs. Phase II applies semantic-preserving mutations to assess the confidence with which LLMs understand code.
• Figure 5: LOC Distribution for the Final Set of Programs