Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Accuracy: Characterizing Code Comprehension Capabilities in (Large) Language Models

Felix Mächtle, Jan-Niclas Serr, Nils Loose, Thomas Eisenbarth

TL;DR

The paper tackles the problem that aggregate benchmarks fail to reveal task-level nuances in LLM code understanding. It introduces a binary input–output consistency framework and a large CodeNet-based Python dataset to compare human-centric static code metrics against a feature-free shadow predictor trained on raw inputs. Key findings show that traditional metrics correlate only weakly with LLM success (AUROC ≈ 0.63), while shadow models achieve AUROC ≈ 0.86, though substantial uncertainty remains and results are model-specific. The study highlights the need for instance-level diagnostics, suggests limitations in predicting correct outcomes, and advocates hybrid benchmarking approaches that blend interpretable metrics with learned signals for safer and more reliable deployment.

Abstract

Large Language Models (LLMs) are increasingly integrated into software engineering workflows, yet current benchmarks provide only coarse performance summaries that obscure the diverse capabilities and limitations of these models. This paper investigates whether LLMs' code-comprehension performance aligns with traditional human-centric software metrics or instead reflects distinct, non-human regularities. We introduce a diagnostic framework that reframes code understanding as a binary input-output consistency task, enabling the evaluation of classification and generative models. Using a large-scale dataset, we correlate model performance with traditional, human-centric complexity metrics, such as lexical size, control-flow complexity, and abstract syntax tree structure. Our analyses reveal minimal correlation between human-defined metrics and LLM success (AUROC 0.63), while shadow models achieve substantially higher predictive performance (AUROC 0.86), capturing complex, partially predictable patterns beyond traditional software measures. These findings suggest that LLM comprehension reflects model-specific regularities only partially accessible through either human-designed or learned features, emphasizing the need for benchmark methodologies that move beyond aggregate accuracy and toward instance-level diagnostics, while acknowledging fundamental limits in predicting correct outcomes.

Beyond Accuracy: Characterizing Code Comprehension Capabilities in (Large) Language Models

TL;DR

The paper tackles the problem that aggregate benchmarks fail to reveal task-level nuances in LLM code understanding. It introduces a binary input–output consistency framework and a large CodeNet-based Python dataset to compare human-centric static code metrics against a feature-free shadow predictor trained on raw inputs. Key findings show that traditional metrics correlate only weakly with LLM success (AUROC ≈ 0.63), while shadow models achieve AUROC ≈ 0.86, though substantial uncertainty remains and results are model-specific. The study highlights the need for instance-level diagnostics, suggests limitations in predicting correct outcomes, and advocates hybrid benchmarking approaches that blend interpretable metrics with learned signals for safer and more reliable deployment.

Abstract

Large Language Models (LLMs) are increasingly integrated into software engineering workflows, yet current benchmarks provide only coarse performance summaries that obscure the diverse capabilities and limitations of these models. This paper investigates whether LLMs' code-comprehension performance aligns with traditional human-centric software metrics or instead reflects distinct, non-human regularities. We introduce a diagnostic framework that reframes code understanding as a binary input-output consistency task, enabling the evaluation of classification and generative models. Using a large-scale dataset, we correlate model performance with traditional, human-centric complexity metrics, such as lexical size, control-flow complexity, and abstract syntax tree structure. Our analyses reveal minimal correlation between human-defined metrics and LLM success (AUROC 0.63), while shadow models achieve substantially higher predictive performance (AUROC 0.86), capturing complex, partially predictable patterns beyond traditional software measures. These findings suggest that LLM comprehension reflects model-specific regularities only partially accessible through either human-designed or learned features, emphasizing the need for benchmark methodologies that move beyond aggregate accuracy and toward instance-level diagnostics, while acknowledging fundamental limits in predicting correct outcomes.
Paper Structure (17 sections, 3 equations, 1 figure, 1 table)

This paper contains 17 sections, 3 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. CodeNet
  4. (Large) Language Models
  5. SAGE
  6. Methodology
  7. Task formulation and label construction
  8. Human-centric metrics
  9. Predicting LLM success from human metrics
  10. Shadow model (feature-free predictor)
  11. Dataset construction
  12. Experiments
  13. Model Performance
  14. Human Metrics
  15. Non-Human Signal via a Shadow Model
  16. ...and 2 more sections

Figures (1)

  • Figure 1: Overlap of solved benchmark tasks on the verified SWE-bench dataset by the top two open-source models. While overall performance differs minimally (just 4 unique tasks difference), their solved task types substantially diverge, demonstrating differences in model specialization.