Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mitigating Shortcut Reasoning in Language Models: A Gradient-Aware Training Approach

Hongyu Cao, Kunpeng Liu, Dongjie Wang, Yanjie Fu

Abstract

Large language models exhibit strong reasoning capabilities, yet often rely on shortcuts such as surface pattern matching and answer memorization rather than genuine logical inference. We propose Shortcut-Aware Reasoning Training (SART), a gradient-aware framework that detects and mitigates shortcut-promoting samples via ShortcutScore and gradient surgery. Our method identifies shortcut signals through gradient misalignment with validation objectives and answer-token concentration, and modifies training dynamics accordingly. Experiments on controlled reasoning benchmarks show that SART achieves +16.5% accuracy and +40.2% robustness over the strongest baseline, significantly improving generalization under distribution shifts. Code is available at: https://github.com/fuyanjie/short-cut-aware-data-centric-reasoning.

Mitigating Shortcut Reasoning in Language Models: A Gradient-Aware Training Approach

Abstract

Large language models exhibit strong reasoning capabilities, yet often rely on shortcuts such as surface pattern matching and answer memorization rather than genuine logical inference. We propose Shortcut-Aware Reasoning Training (SART), a gradient-aware framework that detects and mitigates shortcut-promoting samples via ShortcutScore and gradient surgery. Our method identifies shortcut signals through gradient misalignment with validation objectives and answer-token concentration, and modifies training dynamics accordingly. Experiments on controlled reasoning benchmarks show that SART achieves +16.5% accuracy and +40.2% robustness over the strongest baseline, significantly improving generalization under distribution shifts. Code is available at: https://github.com/fuyanjie/short-cut-aware-data-centric-reasoning.
Paper Structure (18 sections, 14 equations, 2 figures, 4 tables)

This paper contains 18 sections, 14 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Problem Statement
  3. Overview
  4. ShortcutScore-Based Sample Reweighting
  5. Gradient Surgery for Shortcut Suppression
  6. Experiments
  7. Experimental Setup
  8. Model and Training.
  9. Datasets.
  10. Baselines.
  11. Metrics.
  12. Main Results
  13. Ablation Study
  14. Hyperparameter Sensitivity
  15. ShortcutScore Validation
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Overview of SART. (A) Gradient diagnostics: we compute per-sample gradient alignment with validation gradients and answer-token concentration. (B) ShortcutScore computation combines both signals to identify shortcut-prone samples. (C) Sample reweighting reduces their contribution. (D) Gradient surgery projects harmful components out of updates. (E) Resulting updates prioritize reasoning signals over shortcut patterns.
  • Figure 2: Empirical validation of ShortcutScore. (a) Score vs. true shortcut rate (Pearson $r=0.67$), showing monotonic correlation. (b) Gradient alignment distribution: shortcut samples (red) skew negative; non-shortcut samples (green) cluster near zero. (c) Exponential reweighting curve with $\lambda=3.0$.