Revisiting the Uniform Information Density Hypothesis in LLM Reasoning Traces
Minju Gwak, Guijin Son, Jaehyung Kim
TL;DR
The paper adapts Uniform Information Density to LLM reasoning by defining stepwise information density via per-step entropy and two UID metrics (local uniformity and global non-uniformity). It shows that correct reasoning often features low global uniformity with high local uniformity, and that UID-guided trace selection improves accuracy on math benchmarks, with gains up to 32% for some models. Across model sizes and task difficulties, UID signals exhibit nuanced patterns: smaller models benefit from local smoothing, larger models leverage global non-uniformity, and harder problems favor local uniformity with global non-uniformity. The work also demonstrates UID as an interpretable lens for tracing reasoning structure and highlights its potential generalizability beyond mathematics, while acknowledging limitations and directions for future research.
Abstract
The Uniform Information Density (UID) hypothesis suggests that effective communication maintains a stable flow of information. In this work, we revisit this principle in the context of large language model (LLM) reasoning traces, asking whether step-level uniformity reflects reasoning quality. To this end, we propose an entropy-based stepwise information density metric and introduce two complementary measures of uniformity, local and global uniformity scores. Across the experiments on six different reasoning benchmarks, we find that step-level uniformity not only provides a strong theoretical lens but also yields practical performance benefits; for example, selecting reasoning traces with more uniform information density at the step-level improves accuracy by 10-32\% relative gains over baselines at AIME2025. Our analysis further reveals that correct reasoning traces tend to avoid sharp information density spikes, while incorrect traces exhibit irregular information bursts. These results demonstrate that UID-inspired information density measures outperform alternative internal signals as predictors of reasoning quality. Results highlight the uniformity of the information density as a robust diagnostic and selection criterion for building more reliable and accurate reasoning systems.