Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fragile Reasoning: A Mechanistic Analysis of LLM Sensitivity to Meaning-Preserving Perturbations

Shou-Tzu Han, Rodrigue Rizk, KC Santosh

Abstract

Large language models demonstrate strong performance on mathematical reasoning benchmarks, yet remain surprisingly fragile to meaning-preserving surface perturbations. We systematically evaluate three open-weight LLMs, Mistral-7B, Llama-3-8B, and Qwen2.5-7B, on 677 GSM8K problems paired with semantically equivalent variants generated through name substitution and number format paraphrasing. All three models exhibit substantial answer-flip rates (28.8%-45.1%), with number paraphrasing consistently more disruptive than name swaps. To trace the mechanistic basis of these failures, we introduce the Mechanistic Perturbation Diagnostics (MPD) framework, combining logit lens analysis, activation patching, component ablation, and the Cascading Amplification Index (CAI) into a unified diagnostic pipeline. CAI, a novel metric quantifying layer-wise divergence amplification, outperforms first divergence layer as a failure predictor for two of three architectures (AUC up to 0.679). Logit lens reveals that flipped samples diverge from correct predictions at significantly earlier layers than stable samples. Activation patching reveals a stark architectural divide in failure localizability: Llama-3 failures are recoverable by patching at specific layers (43/60 samples), while Mistral and Qwen failures are broadly distributed (3/60 and 0/60). Based on these diagnostic signals, we propose a mechanistic failure taxonomy (localized, distributed, and entangled) and validate it through targeted repair experiments: steering vectors and layer fine-tuning recover 12.2% of localized failures (Llama-3) but only 7.2% of entangled (Qwen) and 5.2% of distributed (Mistral) failures.

Fragile Reasoning: A Mechanistic Analysis of LLM Sensitivity to Meaning-Preserving Perturbations

Abstract

Large language models demonstrate strong performance on mathematical reasoning benchmarks, yet remain surprisingly fragile to meaning-preserving surface perturbations. We systematically evaluate three open-weight LLMs, Mistral-7B, Llama-3-8B, and Qwen2.5-7B, on 677 GSM8K problems paired with semantically equivalent variants generated through name substitution and number format paraphrasing. All three models exhibit substantial answer-flip rates (28.8%-45.1%), with number paraphrasing consistently more disruptive than name swaps. To trace the mechanistic basis of these failures, we introduce the Mechanistic Perturbation Diagnostics (MPD) framework, combining logit lens analysis, activation patching, component ablation, and the Cascading Amplification Index (CAI) into a unified diagnostic pipeline. CAI, a novel metric quantifying layer-wise divergence amplification, outperforms first divergence layer as a failure predictor for two of three architectures (AUC up to 0.679). Logit lens reveals that flipped samples diverge from correct predictions at significantly earlier layers than stable samples. Activation patching reveals a stark architectural divide in failure localizability: Llama-3 failures are recoverable by patching at specific layers (43/60 samples), while Mistral and Qwen failures are broadly distributed (3/60 and 0/60). Based on these diagnostic signals, we propose a mechanistic failure taxonomy (localized, distributed, and entangled) and validate it through targeted repair experiments: steering vectors and layer fine-tuning recover 12.2% of localized failures (Llama-3) but only 7.2% of entangled (Qwen) and 5.2% of distributed (Mistral) failures.

Paper Structure

This paper contains 33 sections, 5 equations, 6 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Robustness of LLM Reasoning.
  4. Mechanistic Interpretability.
  5. Mathematical Reasoning in LLMs.
  6. Methodology
  7. Perturbation Dataset Construction
  8. Paired Robustness Evaluation
  9. Mechanistic Analysis
  10. Logit Lens Analysis
  11. Activation Patching
  12. Component Ablation
  13. Cascading Amplification Index
  14. Experimental Setup
  15. Results
  16. ...and 18 more sections

Figures (6)

  • Figure 1: Overview of MPD. We construct meaning-preserving perturbation pairs from GSM8K, evaluate paired robustness across models, and diagnose failures with logit lens, activation patching, component ablation, and the Cascading Amplification Index (CAI).
  • Figure 2: Logit lens applied to a flipped sample and a stable sample from Mistral-7B jiang2023mistral7b. JSD rises sharply for the flip case but remains near zero for the stable case.
  • Figure 3: Distribution of first divergence layers for flipped vs. non-flipped samples.
  • Figure 4: Activation patching recovery rate by layer.
  • Figure 5: Per-layer recovery under component ablation. Mistral and Llama-3 show attention dominance, while Qwen shows MLP dominance.
  • ...and 1 more figures