Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

InfoDensity: Rewarding Information-Dense Traces for Efficient Reasoning

Chengwei Wei, Jung-jae Kim, Longyin Zhang, Shengkai Chen, Nancy F. Chen

Abstract

Large Language Models (LLMs) with extended reasoning capabilities often generate verbose and redundant reasoning traces, incurring unnecessary computational cost. While existing reinforcement learning approaches address this by optimizing final response length, they neglect the quality of intermediate reasoning steps, leaving models vulnerable to reward hacking. We argue that verbosity is not merely a length problem, but a symptom of poor intermediate reasoning quality. To investigate this, we conduct an empirical study tracking the conditional entropy of the answer distribution across reasoning steps. We find that high-quality reasoning traces exhibit two consistent properties: low uncertainty convergence and monotonic progress. These findings suggest that high-quality reasoning traces are informationally dense, that is, each step contributes meaningful entropy reduction relative to the total reasoning length. Motivated by this, we propose InfoDensity, a reward framework for RL training that combines an AUC-based reward and a monotonicity reward as a unified measure of reasoning quality, weighted by a length scaling term that favors achieving equivalent quality more concisely. Experiments on mathematical reasoning benchmarks demonstrate that InfoDensity matches or surpasses state-of-the-art baselines in accuracy while significantly reducing token usage, achieving a strong accuracy-efficiency trade-off.

InfoDensity: Rewarding Information-Dense Traces for Efficient Reasoning

Abstract

Large Language Models (LLMs) with extended reasoning capabilities often generate verbose and redundant reasoning traces, incurring unnecessary computational cost. While existing reinforcement learning approaches address this by optimizing final response length, they neglect the quality of intermediate reasoning steps, leaving models vulnerable to reward hacking. We argue that verbosity is not merely a length problem, but a symptom of poor intermediate reasoning quality. To investigate this, we conduct an empirical study tracking the conditional entropy of the answer distribution across reasoning steps. We find that high-quality reasoning traces exhibit two consistent properties: low uncertainty convergence and monotonic progress. These findings suggest that high-quality reasoning traces are informationally dense, that is, each step contributes meaningful entropy reduction relative to the total reasoning length. Motivated by this, we propose InfoDensity, a reward framework for RL training that combines an AUC-based reward and a monotonicity reward as a unified measure of reasoning quality, weighted by a length scaling term that favors achieving equivalent quality more concisely. Experiments on mathematical reasoning benchmarks demonstrate that InfoDensity matches or surpasses state-of-the-art baselines in accuracy while significantly reducing token usage, achieving a strong accuracy-efficiency trade-off.
Paper Structure (34 sections, 7 equations, 9 figures, 2 tables)

This paper contains 34 sections, 7 equations, 9 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Efficient Reasoning.
  4. Evaluation of reasoning traces.
  5. Information-theoretic Analysis of Reasoning Traces
  6. Preliminary
  7. Step-level and Trajectory-level Analysis
  8. Dataset.
  9. Challenges of step-level information gain.
  10. From local steps to full trajectories.
  11. Characterizing high-quality reasoning.
  12. InfoDensity
  13. Information Quality.
  14. Length Scaling.
  15. The InfoDensity Formulation.
  16. ...and 19 more sections

Figures (9)

  • Figure 1: Step-level information gain distributions for correct steps (blue) and first incorrect steps (red) in the ProcessBench subset across four models. Dotted vertical lines mark the group means.
  • Figure 2: ROC curves for step-level information gain as a binary error detector (correct vs. first incorrect step) in the ProcessBench subset across four models.
  • Figure 3: Mean conditional entropy trajectory of Qwen3-4B-Instruct across three datasets: GSM8K, OlympiadBench, and OmniMath. Each curve shows the mean entropy over normalized reasoning steps, with shaded regions indicating $\pm 1$ standard deviation.
  • Figure 4: Accuracy vs. Length Trade-off. This plot illustrates the relationship between mean response length and test accuracy on the GSM8K dataset. Sampling temperature is set to 0 for deterministic generation.
  • Figure 5: Accuracy over training steps for different coefficients $\alpha$ in $R_{\mathrm{quality}}$.
  • ...and 4 more figures