Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting the UID Hypothesis in LLM Reasoning Traces

Minju Gwak, Guijin Son, Jaehyung Kim

TL;DR

The paper investigates whether Uniform Information Density (UID) holds for LLM reasoning traces by introducing entropy-based, per-step metrics. It operationalizes UID through per-step information density $ID_i$ derived from token-level probabilities, and analyzes three math benchmarks using Variance, Gini, and Shannon evenness as global UID metrics. Across AIME\ 2025, HMMT\ 2025, and Minerva Math, the authors find that correct reasoning correlates with globally non-uniform information distribution, with Variance consistently the strongest predictor, and model scaling amplifying this effect. These findings challenge the assumption that stable, uniform information flow underlies machine reasoning and point toward adaptive, non-uniform information patterns as a pathway to more robust and interpretable reasoning.

Abstract

Large language models (LLMs) often solve problems using step-by-step Chain-of-Thought (CoT) reasoning, yet these intermediate steps are frequently unfaithful or hard to interpret. Inspired by the Uniform Information Density (UID) hypothesis in psycholinguistics -- which posits that humans communicate by maintaining a stable flow of information -- we introduce entropy-based metrics to analyze the information flow within reasoning traces. Surprisingly, across three challenging mathematical benchmarks, we find that successful reasoning in LLMs is globally non-uniform: correct solutions are characterized by uneven swings in information density, in stark contrast to human communication patterns. This result challenges assumptions about machine reasoning and suggests new directions for designing interpretable and adaptive reasoning models.

Revisiting the UID Hypothesis in LLM Reasoning Traces

TL;DR

The paper investigates whether Uniform Information Density (UID) holds for LLM reasoning traces by introducing entropy-based, per-step metrics. It operationalizes UID through per-step information density derived from token-level probabilities, and analyzes three math benchmarks using Variance, Gini, and Shannon evenness as global UID metrics. Across AIME\ 2025, HMMT\ 2025, and Minerva Math, the authors find that correct reasoning correlates with globally non-uniform information distribution, with Variance consistently the strongest predictor, and model scaling amplifying this effect. These findings challenge the assumption that stable, uniform information flow underlies machine reasoning and point toward adaptive, non-uniform information patterns as a pathway to more robust and interpretable reasoning.

Abstract

Large language models (LLMs) often solve problems using step-by-step Chain-of-Thought (CoT) reasoning, yet these intermediate steps are frequently unfaithful or hard to interpret. Inspired by the Uniform Information Density (UID) hypothesis in psycholinguistics -- which posits that humans communicate by maintaining a stable flow of information -- we introduce entropy-based metrics to analyze the information flow within reasoning traces. Surprisingly, across three challenging mathematical benchmarks, we find that successful reasoning in LLMs is globally non-uniform: correct solutions are characterized by uneven swings in information density, in stark contrast to human communication patterns. This result challenges assumptions about machine reasoning and suggests new directions for designing interpretable and adaptive reasoning models.
Paper Structure (34 sections, 16 equations, 1 figure, 3 tables)

This paper contains 34 sections, 16 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Fragility of CoT and the role of individual reasoning steps
  4. Intrinsic signals in LLM reasoning
  5. Exploring the UID Hypothesis in Reasoning Models
  6. Background: Uniform information density hypothesis
  7. Preliminary analysis with per-step information density scores of reasoning traces
  8. Three metrics of $ID_i$
  9. Interpretation of the three metrics of $ID_i$ across a reasoning trace
  10. Measuring the uniformity of information density in reasoning trace
  11. Unlike Human Communication, Global Non-uniform Information Distribution Predicts Reasoning Success in LLMs
  12. Implementation Details
  13. Hyperparameters and GPU Setup.
  14. Experiment Setup
  15. Evaluation and Benchmarks
  16. ...and 19 more sections

Figures (1)

  • Figure 1: Averaged $ID$ scores ranging of correct and incorrect traces on AIME2025 test set tracked with step-level information density.