LARFT: Closing the Cognition-Action Gap for Length Instruction Following in Large Language Models

Abstract

Despite the strong performance of Large Language Models (LLMs) on complex instruction-following tasks, precise control of output length remains a persistent challenge. Existing methods primarily attempt to enforce length constraints by externally imposing length signals or optimization objectives, while largely overlooking the underlying limitation: the model's intrinsic deficit in length cognition. To address this, we propose LARFT (Length-Aware Reinforcement Fine-Tuning), a training framework that aligns the model's length cognition with its action. Specifically, LARFT integrates length-oriented reinforcement learning with a hindsight length awareness. By transforming on-policy data into hindsight self-awareness tasks where the model learns to identify the actual length of its own generation, LARFT jointly optimizes the model's internal representation of length information and refines its policy to satisfy length constraints, thereby achieving precise and reliable length instruction following. Extensive experiments across four base models demonstrate that LARFT outperforms existing baselines, achieving an average improvement of +20.92 points across three length instruction following benchmarks with only a marginal decline of -1.45 points on four general capability benchmarks.

Figures (6)

• Figure 1: Overview of the Length-Aware Reinforcement Fine-Tuning (LARFT) framework. (I) Illustrates the Cognition-Action Gap, where standard LLMs fail to align their output length with user instructions. (II) Presents our LARFT method, which unifies Length-Oriented RL and Hindsight Length Awareness to align generation with constraints. (III) Demonstrates that LARFT achieves precise length control while maintaining robust general capabilities.
• Figure 2: (Top) Comparison of length following (SFT) and explicit length awareness (SFT) against the original baseline. (Bottom) Comparison of standard RL, RL initialized with length awareness, and LARFT.
• Figure 3: Ablation study of the unified coefficient $\lambda$. (Left) The plot shows the training reward over steps. (Right) The plot shows the entropy changes. Default setting $\lambda=0.01$ demonstrates superior performance, while "w/o Scheduling" indicates the removal of the Cosine Annealing schedule.
• Figure 4: Length probing correlations across layers. The plots compare the ability to decode response length from the hidden states of the first token (Left) and the last token (Right).
• Figure 5: Cosine annealing schedule for $\lambda_t$.