Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Predicting LLM Output Length via Entropy-Guided Representations

Huanyi Xie, Yubin Chen, Liangyu Wang, Lijie Hu, Di Wang

TL;DR

The paper tackles inefficient batched LLM inference caused by long-tail output lengths by reusing the model's hidden states for length prediction. It introduces Entropy-Guided Token Pooling (EGTP) for static length estimates and Progressive Length Prediction (PLP) for dynamic, stepwise forecasts during decoding, validated on the ForeLen benchmark that includes long sequences, CoT, and RL scenarios. EGTP achieves state-of-the-art accuracy, reducing MAE by about 29.16% over the best baseline and enabling substantial end-to-end throughput gains when paired with a length-aware scheduler. The work demonstrates that critical signals for output length are embedded in the LLM's activations, offering a lightweight, generalizable approach to efficient LLM inference and providing a new evaluation baseline for length prediction in complex generation settings.

Abstract

The long-tailed distribution of sequence lengths in LLM serving and reinforcement learning (RL) sampling causes significant computational waste due to excessive padding in batched inference. Existing methods rely on auxiliary models for static length prediction, but they incur high overhead, generalize poorly, and fail in stochastic "one-to-many" sampling scenarios. We introduce a lightweight framework that reuses the main model's internal hidden states for efficient length prediction. Our framework features two core components: 1) Entropy-Guided Token Pooling (EGTP), which uses on-the-fly activations and token entropy for highly accurate static prediction with negligible cost, and 2) Progressive Length Prediction (PLP), which dynamically estimates the remaining length at each decoding step to handle stochastic generation. To validate our approach, we build and release ForeLen, a comprehensive benchmark with long-sequence, Chain-of-Thought, and RL data. On ForeLen, EGTP achieves state-of-the-art accuracy, reducing MAE by 29.16\% over the best baseline. Integrating our methods with a length-aware scheduler yields significant end-to-end throughput gains. Our work provides a new technical and evaluation baseline for efficient LLM inference.

Predicting LLM Output Length via Entropy-Guided Representations

TL;DR

The paper tackles inefficient batched LLM inference caused by long-tail output lengths by reusing the model's hidden states for length prediction. It introduces Entropy-Guided Token Pooling (EGTP) for static length estimates and Progressive Length Prediction (PLP) for dynamic, stepwise forecasts during decoding, validated on the ForeLen benchmark that includes long sequences, CoT, and RL scenarios. EGTP achieves state-of-the-art accuracy, reducing MAE by about 29.16% over the best baseline and enabling substantial end-to-end throughput gains when paired with a length-aware scheduler. The work demonstrates that critical signals for output length are embedded in the LLM's activations, offering a lightweight, generalizable approach to efficient LLM inference and providing a new evaluation baseline for length prediction in complex generation settings.

Abstract

The long-tailed distribution of sequence lengths in LLM serving and reinforcement learning (RL) sampling causes significant computational waste due to excessive padding in batched inference. Existing methods rely on auxiliary models for static length prediction, but they incur high overhead, generalize poorly, and fail in stochastic "one-to-many" sampling scenarios. We introduce a lightweight framework that reuses the main model's internal hidden states for efficient length prediction. Our framework features two core components: 1) Entropy-Guided Token Pooling (EGTP), which uses on-the-fly activations and token entropy for highly accurate static prediction with negligible cost, and 2) Progressive Length Prediction (PLP), which dynamically estimates the remaining length at each decoding step to handle stochastic generation. To validate our approach, we build and release ForeLen, a comprehensive benchmark with long-sequence, Chain-of-Thought, and RL data. On ForeLen, EGTP achieves state-of-the-art accuracy, reducing MAE by 29.16\% over the best baseline. Integrating our methods with a length-aware scheduler yields significant end-to-end throughput gains. Our work provides a new technical and evaluation baseline for efficient LLM inference.
Paper Structure (39 sections, 9 equations, 13 figures, 11 tables)

This paper contains 39 sections, 9 equations, 13 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Entropy-Guided Token Pooling (EGTP)
  5. Motivation.
  6. EGTP.
  7. Length Regression via Soft Label Distribution
  8. Progressive Length Prediction (PLP)
  9. Experiments
  10. Dataset construction
  11. Scenario 1: Long-Sequence and Complex Reasoning Generation.
  12. Scenario 2: Dynamic RL Sampling.
  13. Data Splits and Statistics.
  14. Experiment Setting
  15. Main Results: Prediction Accuracy
  16. ...and 24 more sections

Figures (13)

  • Figure 1: Analysis of LLM Sequence Length Distributions in Representative Scenarios. Figure (a) illustrates the length characteristics of our proposed ForeLen benchmark and LMSYS, demonstrating a wider and longer-tailed distribution. Figure (b) showcases the high length variance for generations from identical prompts in an online RL training setting.
  • Figure 2: Empirical validation of the relationship between token entropy and importance. (Left) The box chart shows the average importance for tokens binned into five equal-width intervals based on their entropy value. This indicates that importance increases with entropy. (Right) The scatter plot displays the joint distribution of entropy and importance, with the regression line confirming a significant positive correlation (r=0.451).
  • Figure 3: MAE improvements for Progressive Length Prediction.
  • Figure 4: Proportional distribution of unique prompts across the three primary scenarios.
  • Figure 5: Prompt length distributions across different scenarios.
  • ...and 8 more figures