Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When AI Shows Its Work, Is It Actually Working? Step-Level Evaluation Reveals Frontier Language Models Frequently Bypass Their Own Reasoning

Abhinaba Basu, Pavan Chakraborty

Abstract

Language models increasingly "show their work" by writing step-by-step reasoning before answering. But are these reasoning steps genuinely used, or decorative narratives generated after the model has already decided? Consider: a medical AI writes "The patient's eosinophilia and livedo reticularis following catheterization suggest cholesterol embolization syndrome. Answer: B." If we remove the eosinophilia observation, does the diagnosis change? For most frontier models, the answer is no - the step was decorative. We introduce step-level evaluation: remove one reasoning sentence at a time and check whether the answer changes. This simple test requires only API access -- no model weights -- and costs approximately $1-2 per model per task. Testing 10 frontier models (GPT-5.4, Claude Opus, DeepSeek-V3.2, MiniMax-M2.5, Kimi-K2.5, and others) across sentiment, mathematics, topic classification, and medical QA (N=376-500 each), the majority produce decorative reasoning: removing any step changes the answer less than 17% of the time, while any single step alone recovers the answer. This holds even on math, where smaller models (0.8-8B) show genuine step dependence (55% necessity). Two models break the pattern: MiniMax-M2.5 on sentiment (37% necessity) and Kimi-K2.5 on topic classification (39%) - but both shortcut other tasks. Faithfulness is model-specific and task-specific. We also discover "output rigidity": on the same medical questions, Claude Opus writes 11 diagnostic steps while GPT-OSS-120B outputs a single token. Mechanistic analysis (attention patterns) confirms that CoT attention drops more in late layers for decorative tasks (33%) than faithful ones (20%). Implications: step-by-step explanations from frontier models are largely decorative, per-model per-domain evaluation is essential, and training objectives - not scale - determine whether reasoning is genuine.

When AI Shows Its Work, Is It Actually Working? Step-Level Evaluation Reveals Frontier Language Models Frequently Bypass Their Own Reasoning

Abstract

Language models increasingly "show their work" by writing step-by-step reasoning before answering. But are these reasoning steps genuinely used, or decorative narratives generated after the model has already decided? Consider: a medical AI writes "The patient's eosinophilia and livedo reticularis following catheterization suggest cholesterol embolization syndrome. Answer: B." If we remove the eosinophilia observation, does the diagnosis change? For most frontier models, the answer is no - the step was decorative. We introduce step-level evaluation: remove one reasoning sentence at a time and check whether the answer changes. This simple test requires only API access -- no model weights -- and costs approximately $1-2 per model per task. Testing 10 frontier models (GPT-5.4, Claude Opus, DeepSeek-V3.2, MiniMax-M2.5, Kimi-K2.5, and others) across sentiment, mathematics, topic classification, and medical QA (N=376-500 each), the majority produce decorative reasoning: removing any step changes the answer less than 17% of the time, while any single step alone recovers the answer. This holds even on math, where smaller models (0.8-8B) show genuine step dependence (55% necessity). Two models break the pattern: MiniMax-M2.5 on sentiment (37% necessity) and Kimi-K2.5 on topic classification (39%) - but both shortcut other tasks. Faithfulness is model-specific and task-specific. We also discover "output rigidity": on the same medical questions, Claude Opus writes 11 diagnostic steps while GPT-OSS-120B outputs a single token. Mechanistic analysis (attention patterns) confirms that CoT attention drops more in late layers for decorative tasks (33%) than faithful ones (20%). Implications: step-by-step explanations from frontier models are largely decorative, per-model per-domain evaluation is essential, and training objectives - not scale - determine whether reasoning is genuine.
Paper Structure (49 sections, 3 figures, 6 tables)

This paper contains 49 sections, 3 figures, 6 tables.

Table of Contents

  1. The problem with AI explanations
  2. A simple test anyone can run
  3. Why these three tests together?
  4. Relation to prior work.
  5. Taxonomy.
  6. Ten models, four tasks, one dominant pattern
  7. The dominant pattern: decorative reasoning
  8. The exceptions: MiniMax-M2.5 and Kimi-K2.5
  9. The scale inversion: small models reason more faithfully on math
  10. Full results across four domains
  11. The models that refuse to explain
  12. A tale of two responses
  13. The methodological paradox
  14. Output rigidity across the spectrum
  15. Discussion
  16. ...and 34 more sections

Figures (3)

  • Figure 1: Step-level evaluation illustrated on a sentiment example.Top: The model produces a 3-step reasoning chain. Middle (Necessity): We remove Step 1 and check if the answer changes. If it doesn't, Step 1 was not necessary. Bottom (Sufficiency): We present Step 2 alone. If the model still says "positive," Step 2 is individually sufficient. A faithful model should show high necessity (removing steps hurts) and low sufficiency (no single step is enough).
  • Figure 2: Step necessity across 10 models. Only MiniMax-M2.5 crosses the 30% faithfulness threshold (dashed line) on both tasks. All other models cluster below 17%. Missing GSM8K bars indicate insufficient multi-step data for that model.
  • Figure 3: Output rigidity varies by model and task. Each bar shows the percentage of 500 examples where the model produced ${\geq}2$ reasoning steps. Claude Opus and DeepSeek almost always explain; Qwen3.5-397B almost never does. GPT-OSS shows the starkest task dependence: 99--100% on classification but only 38% on medical QA. Missing bars (height 0) indicate the task was not yet evaluated.