Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Reasoning

TL;DR

This work distinguishes lexical recall (verbatim code retrieval) from semantic recall (understanding code behavior) in long-context code understanding. It introduces semantic recall sensitivity as a benchmark property and a counterfactual line-removal method to measure it, revealing that existing benchmarks often mask semantic failures through pattern matching. The SemTrace task is proposed to force semantic recall by using unpredictable operations, uncovering pronounced position-dependent degradation (median $92.73\%$) that far exceeds CRUXEval's $53.36\%$ on existing tasks. The findings suggest current evaluations substantially underestimate semantic recall fragility in long-context code understanding and motivate high-sensitivity benchmarks and careful interpretation for production deployments. Overall, the paper highlights a critical gap between observed lexical competence and genuine semantic understanding in state-of-the-art models as context length grows.

Abstract

Large language models (LLMs) are increasingly deployed for understanding large codebases, but whether they understand operational semantics of long code context or rely on pattern matching shortcuts remains unclear. We distinguish between lexical recall (retrieving code verbatim) and semantic recall (understanding operational semantics). Evaluating 10 state-of-the-art LLMs, we find that while frontier models achieve near-perfect, position-independent lexical recall, semantic recall degrades severely when code is centrally positioned in long contexts. We introduce semantic recall sensitivity to measure whether tasks require understanding of code's operational semantics vs. permit pattern matching shortcuts. Through a novel counterfactual measurement method, we show that models rely heavily on pattern matching shortcuts to solve existing code understanding benchmarks. We propose a new task SemTrace, which achieves high semantic recall sensitivity through unpredictable operations; LLMs' accuracy exhibits severe positional effects, with median accuracy drops of 92.73% versus CRUXEval's 53.36% as the relevant code snippet approaches the middle of the input code context. Our findings suggest current evaluations substantially underestimate semantic recall failures in long context code understanding.

Figures (12)

• Figure 1: Python interpreter performance on incomplete CRUXEval functions. Accuracy drops sharply and approaches zero beyond 20% removal, establishing the expected pattern when output prediction genuinely requires understanding all provided code. This provides a reference for evaluating LLM semantic recall sensitivity. Exponential trend fitted with bootstrapped 95% CI.
• Figure 2: Dissociation between lexical and semantic recall across code positions in long contexts. (a) Lexical recall (function retrieval) remains near-perfect and position-independent across all context lengths for frontier models. (b) Semantic recall on CRUXEval output prediction (CRUXEval-O) exhibits moderate lost-in-the-middle effects, with median accuracy decreases of 53.36% as context length increases and target code moves toward the middle. (c) High-sensitivity semantic recall (SemTrace) shows severe position dependence, with median accuracy plummeting up to 92.73% when target code is centrally positioned.
• Figure 3: CRUXEval semantic recall sensitivity measurement across all models. Performance on input (blue) and output (orange) prediction tasks as lines are progressively removed from functions. We fit exponential trend lines using non-linear least squares and and leverage bootstrapping to compute 95% confidence intervals. Even with 50% of lines removed, models' accuracy drops only 44.15% and 59.74% for input and output prediction, respectively, showing gradual degradation that contrasts sharply with the Python interpreter's exponential decay (\ref{['fig: incomp sensitivity python']}). This gradual degradation indicates low semantic recall sensitivity, meaning models compensate for missing code through pattern matching from pretraining rather than relying on semantic understanding of the specific provided code.
• Figure 4: GPT 4.1's higher-digit SemTrace performance reveals arithmetic memorization. Testing with 2-6 digit operations at 80 distractors shows position-independent performance for 2-3 digits, but clear position-dependent degradation for 4+ digits (31pp drop at 4 digits, 43% relative loss at 6 digits). This demonstrates GPT 4.1's perfect 2-digit performance (\ref{['fig: long context panel']}c) reflects memorized arithmetic rather than superior semantic recall.
• Figure 5: Dissociation between lexical and semantic recall across code positions in long contexts for smaller models. Positional effects across lexical recall (a), semantic recall on CRUXEval output prediction (CRUXEval-O) (b), and high-sensitivity semantic recall on SemTrace (c). Despite lower absolute performance, smaller models exhibit the same fundamental dissociation as frontier models: position-independent lexical recall and position-dependent semantic recall, with more severe degradation on SemTrace.