"I May Not Have Articulated Myself Clearly": Diagnosing Dynamic Instability in LLM Reasoning at Inference Time

TL;DR

This work introduces an inference-time diagnostic for dynamic instability in LLM reasoning that uses only black-box, per-step top-$k$ token distributions. It defines an instability score $I_t = D_t + \lambda H_t$ with $D_t = \mathrm{JSD}(\tilde{p}_t,\tilde{p}_{t-1})$ and $H_t$ the entropy, and summarizes global instability as $S = \max_t I_t$, while also leveraging peak timing $\rho$ to distinguish corrective from destructive instability. Across GSM8K and HotpotQA, the instability strength correlates with failure rate and can separate correct from incorrect traces, with robust results across models and decoding settings; importantly, early instability tends to be recoverable while late instability is more destructive, highlighting the role of decoding horizon. The method is training-free, model-agnostic, and reproducible, offering a diagnostic lens rather than a control mechanism, and it emphasizes that instability can be informative rather than uniformly harmful. These findings contribute to understanding reasoning dynamics in LLMs and suggest timing-aware analysis as a principled approach to diagnosing trajectory-level failures.

Abstract

Reasoning failures in large language models (LLMs) are typically measured only at the end of a generation, yet many failures manifest as a process-level breakdown: the model "loses the thread" mid-reasoning. We study whether such breakdowns are detectable from inference-time observables available in standard APIs (token log probabilities), without any training or fine-tuning. We define a simple instability signal that combines consecutive-step distributional shift (JSD) and uncertainty (entropy), summarize each trace by its peak instability strength, and show that this signal reliably predicts failure. Across GSM8K and HotpotQA, instability strength predicts wrong answers with above-chance AUC and yields monotonic bucket-level accuracy decline at scale across model sizes. Crucially, we show that instability is not uniformly harmful: early instability can reflect subsequent stabilization and a correct final answer (\emph{corrective instability}), whereas late instability is more often followed by failure (\emph{destructive instability}), even at comparable peak magnitudes, indicating that recoverability depends not only on how strongly the distribution changes but also on when such changes occur relative to the remaining decoding horizon. The method is model-agnostic, training-free, and reproducible, and is presented as a diagnostic lens rather than a corrective or control mechanism.

Figures (12)

• Figure 1: Inference-time dynamic instability as a diagnostic signal. Conceptual overview of an inference-time diagnostic for dynamic instability in large language model reasoning. A: At each decoding step $t$, we observe only the renormalized top-$k$ next-token distribution $\tilde{p}_t$, which is accessible as a black-box inference-time signal; no hidden states, gradients, or training access are required. B: An instability event is characterized by an abrupt reshuffling of probability mass among high-probability candidates, accompanied by increased uncertainty (higher entropy) between consecutive steps. C: We define a diagnostic instability signal $I_t = D_t + \lambda H_t$, combining distributional shift $D_t = \mathrm{JSD}(\tilde{p}_t,\tilde{p}_{t-1})$ and uncertainty $H_t$. All quantities are computed solely from $\tilde{p}_t$ at inference time, with overall instability strength summarized as $S = \max_t I_t$. D: Instability strength $S$ is associated with increased failure risk, while the peak location (relative position $\rho$) provides complementary information, separating earlier (more recoverable/corrective) from later (less recoverable/destructive) instability episodes.
• Figure 2: Timing-dependent regimes of inference-time instability. Representative decoding traces illustrating how the timing of instability, rather than its magnitude alone, is associated with different reasoning outcomes. A: A representative trace with an early instability peak (recoverable / corrective). Although the instability signal $I_t$ reaches a high value, it occurs sufficiently early in decoding for the model to recover, resulting in a correct final answer. B: A representative trace with a late instability peak (unrecoverable / destructive). Despite comparable instability strength, the peak occurs near the end of decoding, leaving limited opportunity for recovery and resulting in an incorrect final answer. C: A controlled comparison with instability signals normalized to comparable peak strength, demonstrating that magnitude alone is insufficient: similar instability strength can correspond to different outcomes depending on peak timing. All curves show representative (not averaged) decoding trajectories, with instability defined as $I_t = D_t + \lambda H_t$.
• Figure 3: Monotonic bucket trends for Llama-3.2-1B-Instruct across temperature settings ($\tau \in \{0.0, 0.3, 0.7\}$). Higher instability buckets consistently show lower accuracy, confirming the predictive value of the instability signal.
• Figure 4: Accuracy by peak instability position on the 100-trace held-out baseline run (same setting as \ref{['tab:corrective-destructive']}). Early peaks allow recovery and yield higher accuracy; late peaks leave insufficient time to recover.
• Figure 5: Failure-mode distribution on GSM8K (300 examples) for Llama-3.2-1B-Instruct greedy. We partition wrong traces into three disjoint categories for interpretability: stable wrong (wrong traces in the lowest quintile of instability strength $S$), early collapse (wrong traces in the highest quintile of early-window strength $S_{20}=\max_{t\le 20} I_t$), and unstable wrong (the remaining wrong traces). This breakdown highlights that a non-trivial fraction of errors are stable wrong, consistent with the limitation that instability is not a universal explanation of failure.

Theorems & Definitions (12)

• Definition 1: Finite-horizon stability
• Definition 2: Local expansion rate
• Lemma 1: Hidden-state change induces projected logit change
• Lemma 2: Observable distributional change reflects logit change
• Proposition 1: Top-$k$ analogue (restricted simplex)
• Theorem 1: Observable instability lower-bounds internal step change
• Definition 3: Decision margin
• Lemma 3: Margin robustness
• Lemma 4: Entropy upper-bounds peak confidence
• Theorem 2: Late correction penalty