EDIS: Diagnosing LLM Reasoning via Entropy Dynamics

TL;DR

This work tackles the challenge of distinguishing correct LLM reasoning from plausible yet incorrect outputs by analyzing the temporal evolution of token-level entropy rather than relying on static confidence. It introduces Entropy Dynamics Instability Score (EDIS), a trajectory-level metric that combines burst spikes and peak-valley spikes to quantify instability in entropy dynamics during generation. Empirical results show that EDIS enables substantial improvements in inference-time selection (up to an 82% relative gain across benchmarks) and provides promising signals for reinforcement learning data curation, outperforming mean entropy and other confidence measures. The findings suggest entropy dynamics offer a robust, model-agnostic lens for diagnosing and improving LLM reasoning, with practical impact on both deployment-time filtering and training-time sample selection.

Abstract

Entropy-based confidence signals are increasingly leveraged to improve reasoning in large language models (LLMs), yet existing approaches treat confidence as a static quantity -- typically aggregated over tokens. We show that the \emph{temporal evolution} of confidence during generation carries richer information than aggregate statistics alone. Analyzing token-level entropy trajectories, we identify characteristic patterns distinguishing correct from incorrect reasoning: erroneous solutions exhibit unstable dynamics, including burst spikes (sustained uncertainty growth) and peak-valley spikes (sharp rebounds following transient confidence). These patterns persist across models and training stages, suggesting they reflect intrinsic properties of reasoning failure rather than superficial noise. To formalize this observation, we introduce the Entropy Dynamics Instability Score (\textbf{EDIS}), a trajectory-level metric quantifying instability in entropy evolution. EDIS serves as an effective diagnostic signal for inference-time selection, substantially improving reasoning accuracy, and offers a promising direction for training-time sample curation. Our findings establish entropy dynamics as an underexplored yet informative lens for understanding and improving LLM reasoning.

Figures (12)

• Figure 1: Token entropy trajectories for correct (top) and incorrect (bottom) reasoning. Correct responses maintain stable, low entropy, while incorrect responses exhibit distinctive instability patterns: peak-valley spikes (entropy drops then rebounds) and burst spikes (progressive entropy rise).
• Figure 2: EDIS (left) vs. mean entropy (right) distributions. EDIS clearly separates correct from incorrect responses, while mean entropy distributions largely overlap.
• Figure 3: EDIS-based sample curation for RL training.
• Figure 4: EDIS-based best-$k$-of-$N$ selection across models and benchmarks ($k = 8$). Accuracy improves consistently as the oversampling multiplier $m$ increases. Results averaged across three temperatures ($0.2$, $0.6$, $1.0$).
• Figure 5: EDIS vs. Mean Entropy as correctness predictors. (a-b) EDIS provides clearer separation. (c) EDIS achieves higher AUC (0.804 vs. 0.673). (d) EDIS maintains advantage across retention rates.