DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learning

TL;DR

This work revisits reinforcement learning for reasoning efficiency and argues that the bottleneck lies more in optimization dynamics than in the length-penalty design itself. By combining batch-wise reward normalization, higher update clipping, dynamic sampling, and a simple truncation penalty, the authors introduce Doing Length pEnalty Right (DLER), achieving state-of-the-art accuracy–length trade-offs and enabling strong test-time scaling through parallel thinking. A difficulty-aware variant (DA-DLER) further tightens truncation on easy questions for additional gains, while an update-selective weight merging strategy offers a training-free path to recover accuracy when using public data. Overall, the results demonstrate that substantial reductions in output length can be achieved without sacrificing performance, with practical implications for deploying concise, accurate reasoning systems at scale.

Abstract

Reasoning language models such as OpenAI-o1, DeepSeek-R1, and Qwen achieve strong performance via extended chains of thought but often generate unnecessarily long outputs. Maximizing intelligence per token--accuracy relative to response length--remains an open problem. We revisit reinforcement learning (RL) with the simplest length penalty--truncation--and show that accuracy degradation arises not from the lack of sophisticated penalties but from inadequate RL optimization. We identify three key challenges: (i) large bias in advantage estimation, (ii) entropy collapse, and (iii) sparse reward signal. We address them with Doing Length pEnalty Right (DLER), a training recipe combining batch-wise reward normalization, higher clipping, dynamic sampling, and a simple truncation length penalty. DLER achieves state-of-the-art accuracy--efficiency trade-offs, cutting output length by over 70 percent while surpassing all previous baseline accuracy. It also improves test-time scaling: compared to DeepSeek-R1-7B, DLER-7B generates multiple concise responses in parallel with 28 percent higher accuracy and lower latency. We further introduce Difficulty-Aware DLER, which adaptively tightens truncation on easier questions for additional efficiency gains. We also propose an update-selective merging method that preserves baseline accuracy while retaining the concise reasoning ability of the DLER model, which is useful for scenarios where RL training data is scarce.

Figures (11)

• Figure 1: (a) DLER achieves state-of-the-art accuracy/length trade-offs, shortening CoT by up to 70% without losing accuracy. (b) On AIME-24, DLER-R1 models enable better test-time scaling. Results for 1.5B models are shown in Fig. \ref{['fig:teaser_appendix']} and Fig. \ref{['fig:parallel_latency_teaser_appendix']} in the appendix.
• Figure 2: Accuracy and average response length of DeepSeek-R1-7B on the AIME-24 test set, evaluated every 10 training steps across two RL training runs: group-wise reward normalization (GRPO) and batch-wise normalization. GRPO shows declining accuracy while batch-wise reward normalization remains stable despite reduced token counts.
• Figure 3: (a) Word clouds of the most frequent tokens clipped by the high-threshold ($1+\epsilon$) before applying higher clipping threshold,showing that many are transitional words important for reasoning, and clipping them limits exploration during RL training.. (b) Average probability and entropy of tokens not clipped, clipped by the higher threshold, and clipped by the lower threshold during RL training. Clipped tokens have much lower probabilities than unclipped ones, and those clipped by the higher threshold consistently show higher entropy, supporting Fig. \ref{['fig:high-clip-word-cloud']} that these are often high-entropy transitional tokens triggering reasoning steps.
• Figure 4: Left: Ratio of training prompts with all 16 rollouts receiving zero reward, including those caused by exceeding the truncation length. Around half of the prompts fall into this category early in training, weakening the signal and biasing the model toward easier prompts that model already know how to solve within the target length. Right: Ratio of training prompts with all 16 rollouts receiving reward score of one steadily increases, while average response length declines and remains markedly shorter than that for prompts whose rollouts all receive a reward of zero.
• Figure 5: Accuracy and average response length of DeepSeek-R1-7B on the AIME-24 test set, evaluated every 10 training steps across two RL training runs: one with batch-wise reward normalization and the other with our DLER recipe. DLER enhances the accuracy–token efficiency by reducing token usage while fully recovering the accuracy loss of applying plain batch-wise reward normalization.