Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reasoning Stabilization Point: A Training-Time Signal for Stable Evidence and Shortcut Reliance

Sahil Rajesh Dhayalkar

TL;DR

The paper addresses how finetuning pretrained language models can subtly shift decision evidence beyond what accuracy captures. It introduces a training-time interpretability framework by treating token-level attributions as a time series and defines explanation drift and the Reasoning Stabilization Point (RSP) to summarize when evidence becomes stable. Empirically, drift stabilizes early across DistilBERT and MiniLM on SST-2 and QNLI, even as accuracy continues to improve, and the drift signal can reveal shortcut reliance under controlled spur injections. This provides a low-cost diagnostic for checkpoint selection and robustness monitoring during fine-tuning.

Abstract

Fine-tuning pretrained language models can improve task performance while subtly altering the evidence a model relies on. We propose a training-time interpretability view that tracks token-level attributions across finetuning epochs. We define explanation driftas the epoch-to-epoch change in normalized token attributions on a fixed probe set, and introduce the Reasoning Stabilization Point(RSP), the earliest epoch after which drift remains consistently low. RSP is computed from within-run drift dynamics and requires no tuning on out-of-distribution data. Across multiple lightweight transformer classifiers and benchmark classification tasks, drift typically collapses into a low, stable regime early in training, while validation accuracy continues to change only marginally. In a controlled shortcut setting with label-correlated trigger tokens, attribution dynamics expose increasing reliance on the shortcut even when validation accuracy remains competitive. Overall, explanation drift provides a simple, low-cost diagnostic for monitoring how decision evidence evolves during fine-tuning and for selecting checkpoints in a stable-evidence regime.

Reasoning Stabilization Point: A Training-Time Signal for Stable Evidence and Shortcut Reliance

TL;DR

The paper addresses how finetuning pretrained language models can subtly shift decision evidence beyond what accuracy captures. It introduces a training-time interpretability framework by treating token-level attributions as a time series and defines explanation drift and the Reasoning Stabilization Point (RSP) to summarize when evidence becomes stable. Empirically, drift stabilizes early across DistilBERT and MiniLM on SST-2 and QNLI, even as accuracy continues to improve, and the drift signal can reveal shortcut reliance under controlled spur injections. This provides a low-cost diagnostic for checkpoint selection and robustness monitoring during fine-tuning.

Abstract

Fine-tuning pretrained language models can improve task performance while subtly altering the evidence a model relies on. We propose a training-time interpretability view that tracks token-level attributions across finetuning epochs. We define explanation driftas the epoch-to-epoch change in normalized token attributions on a fixed probe set, and introduce the Reasoning Stabilization Point(RSP), the earliest epoch after which drift remains consistently low. RSP is computed from within-run drift dynamics and requires no tuning on out-of-distribution data. Across multiple lightweight transformer classifiers and benchmark classification tasks, drift typically collapses into a low, stable regime early in training, while validation accuracy continues to change only marginally. In a controlled shortcut setting with label-correlated trigger tokens, attribution dynamics expose increasing reliance on the shortcut even when validation accuracy remains competitive. Overall, explanation drift provides a simple, low-cost diagnostic for monitoring how decision evidence evolves during fine-tuning and for selecting checkpoints in a stable-evidence regime.
Paper Structure (19 sections, 7 equations, 2 figures, 1 table, 1 algorithm)

This paper contains 19 sections, 7 equations, 2 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Post-hoc explanations in NLP.
  4. Faithfulness, sanity checks, and stability of explanations.
  5. Shortcut learning, dataset artifacts, and training dynamics.
  6. Explanation Drift Across Fine-Tuning Time
  7. Main claim.
  8. Setup and token-level explanations
  9. Explanation drift
  10. Reasoning Stabilization Point (RSP)
  11. Experiments
  12. Experimental setup
  13. Experiment 1: Does explanation stabilize before accuracy?
  14. Experiment 2: Drift reveals shortcut adoption
  15. Conclusion
  16. ...and 4 more sections

Figures (2)

  • Figure 1: Experiment 1 Full Results: Validation accuracy (solid) and explanation drift $D_t$ (dashed) across the two settings. The dotted vertical line marks the Reasoning Stabilization Point (RSP; Eq. \ref{['eq:rsp']}). Accuracy changes only marginally after RSP, consistent with explanations stabilizing by (or before) accuracy saturation.
  • Figure 2: Experiment 2 Full Results: Validation accuracy (solid), drift $D_t$ (dashed), and spur attribution mass $M_t$ (dotted; Eq. 7) across epochs for a representative seed. The spur token is injected into training with label correlation (validation remains clean). $M_t$ increases/peaks near RSP, indicating growing reliance on the shortcut feature despite competitive validation accuracy.