From Abstract to Contextual: What LLMs Still Cannot Do in Mathematics

TL;DR

ContextMATH investigates contextual mathematical reasoning by transforming AIME and MATH-500 problems into Scenario Grounding and Complexity Scaling variants to probe how well LLMs extract and solve the mathematical core from narratives. The study finds large accuracy drops on contextual variants across open-source and proprietary models, with misformulation identified as the dominant failure mode and a second bottleneck in reasoning that persists even as models scale. Formulation accuracy declines with problem difficulty, and while larger models increase sufficiency, them reaching high sufficiency remains challenging, indicating a gap between understanding and execution. End-to-end training with contextual data yields robustness gains, whereas training a dedicated formulation-only model provides little to no improvement, suggesting that integrated approaches are needed to advance both formulation and reasoning for reliable real-world mathematical problem solving.

Abstract

Large language models now solve many benchmark math problems at near-expert levels, yet this progress has not fully translated into reliable performance in real-world applications. We study this gap through contextual mathematical reasoning, where the mathematical core must be formulated from descriptive scenarios. We introduce ContextMATH, a benchmark that repurposes AIME and MATH-500 problems into two contextual settings: Scenario Grounding (SG), which embeds abstract problems into realistic narratives without increasing reasoning complexity, and Complexity Scaling (CS), which transforms explicit conditions into sub-problems to capture how constraints often appear in practice. Evaluating 61 proprietary and open-source models, we observe sharp drops: on average, open-source models decline by 13 and 34 points on SG and CS, while proprietary models drop by 13 and 20. Error analysis shows that errors are dominated by incorrect problem formulation, with formulation accuracy declining as original problem difficulty increases. Correct formulation emerges as a prerequisite for success, and its sufficiency improves with model scale, indicating that larger models advance in both understanding and reasoning. Nevertheless, formulation and reasoning remain two complementary bottlenecks that limit contextual mathematical problem solving. Finally, we find that fine-tuning with scenario data improves performance, whereas formulation-only training is ineffective. However, performance gaps are only partially alleviated, highlighting contextual mathematical reasoning as a central unsolved challenge for LLMs.

Figures (4)

• Figure 1: Example from ContextMATH, based on AIME 2025 Problem 15. In Scenario Grounding (SG), mathematical components are mapped to a narrative. In Complexity Scaling (CS), explicit conditions are concealed in sub-problems requiring an extra inference step. Consistent color-coding highlights correspondence between mathematical components across the three versions. LLMs remain strong on abstract benchmarks but drop accuracy on SG, with the gap widening further on CS.
• Figure 2: Distribution of error types in failure cases on AIME 2024/2025 SG and CS problems, where ratios indicate the proportion of cases exhibiting each error type.
• Figure 3: Relationship between reasoning accuracy and formulation metrics. Each point represents a model, with the x-axis showing its average reasoning accuracy across all subsets and the y-axis showing the corresponding values of formulation accuracy (orange), necessity (green), and sufficiency (blue). The fitted lines indicate the overall trends.
• Figure 4: Average accuracy on ContextMATH. “w/o” denotes directly solving the scenario with the reasoning model.