Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leash: Adaptive Length Penalty and Reward Shaping for Efficient Large Reasoning Model

Yanhao Li, Lu Ma, Jiaran Zhang, Lexiang Tang, Wentao Zhang, Guibo Luo

TL;DR

Leash introduces an adaptive length control framework for large reasoning models by formulating generation length as a constrained optimization problem. It uses a Lagrangian primal–dual scheme with a dual variable $\lambda$ to dynamically adjust a length penalty and a one-sided penalized reward to stabilize training, enabling concise yet accurate reasoning. Across mathematical and general reasoning benchmarks, Leash substantially reduces token usage (around 60% on average) while maintaining or improving task performance and demonstrating robust out-of-domain generalization. This approach offers a practical path toward controllable, compute-efficient reasoning in large language models without manual tuning of penalties.

Abstract

Existing approaches typically rely on fixed length penalties, but such penalties are hard to tune and fail to adapt to the evolving reasoning abilities of LLMs, leading to suboptimal trade-offs between accuracy and conciseness. To address this challenge, we propose Leash (adaptive LEngth penAlty and reward SHaping), a reinforcement learning framework for efficient reasoning in LLMs. We formulate length control as a constrained optimization problem and employ a Lagrangian primal-dual method to dynamically adjust the penalty coefficient. When generations exceed the target length, the penalty is intensified; when they are shorter, it is relaxed. This adaptive mechanism guides models toward producing concise reasoning without sacrificing task performance. Experiments on Deepseek-R1-Distill-Qwen-1.5B and Qwen3-4B-Thinking-2507 show that Leash reduces the average reasoning length by 60% across diverse tasks - including in-distribution mathematical reasoning and out-of-distribution domains such as coding and instruction following - while maintaining competitive performance. Our work thus presents a practical and effective paradigm for developing controllable and efficient LLMs that balance reasoning capabilities with computational budgets.

Leash: Adaptive Length Penalty and Reward Shaping for Efficient Large Reasoning Model

TL;DR

Leash introduces an adaptive length control framework for large reasoning models by formulating generation length as a constrained optimization problem. It uses a Lagrangian primal–dual scheme with a dual variable to dynamically adjust a length penalty and a one-sided penalized reward to stabilize training, enabling concise yet accurate reasoning. Across mathematical and general reasoning benchmarks, Leash substantially reduces token usage (around 60% on average) while maintaining or improving task performance and demonstrating robust out-of-domain generalization. This approach offers a practical path toward controllable, compute-efficient reasoning in large language models without manual tuning of penalties.

Abstract

Existing approaches typically rely on fixed length penalties, but such penalties are hard to tune and fail to adapt to the evolving reasoning abilities of LLMs, leading to suboptimal trade-offs between accuracy and conciseness. To address this challenge, we propose Leash (adaptive LEngth penAlty and reward SHaping), a reinforcement learning framework for efficient reasoning in LLMs. We formulate length control as a constrained optimization problem and employ a Lagrangian primal-dual method to dynamically adjust the penalty coefficient. When generations exceed the target length, the penalty is intensified; when they are shorter, it is relaxed. This adaptive mechanism guides models toward producing concise reasoning without sacrificing task performance. Experiments on Deepseek-R1-Distill-Qwen-1.5B and Qwen3-4B-Thinking-2507 show that Leash reduces the average reasoning length by 60% across diverse tasks - including in-distribution mathematical reasoning and out-of-distribution domains such as coding and instruction following - while maintaining competitive performance. Our work thus presents a practical and effective paradigm for developing controllable and efficient LLMs that balance reasoning capabilities with computational budgets.
Paper Structure (22 sections, 13 equations, 3 figures, 4 tables)

This paper contains 22 sections, 13 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Experiments
  4. Experimental Setup
  5. Models and Datasets.
  6. Training configuration.
  7. Baselines and Variants
  8. Evaluation protocol.
  9. Results and Analysis
  10. Main Results
  11. In-domain reasoning performance.
  12. Out-of-domain generalization.
  13. Behavior over Training
  14. Changes of Thinking Patterns
  15. Related Works
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Training dynamics of Leash and Leash-C under the target length $L_t=4\mathrm{k}$ on the 1.5B base. Left: average tokens per response vs. training steps—Leash shortens trajectories faster and stabilizes at a lower length with smaller drift. Right: average accuracy vs. training steps—both variants remain comparable, indicating that length compression does not degrade task accuracy.
  • Figure 2: Training dynamics of Leash and Leash-C under $L_t = 4\text{k}$. The plots show (from left to right): (1) satisfaction rate on the training set, (2) adaptive penalty coefficient $\lambda$, (3) effective penalty value, and (4) average token length. Leash dynamically adjusts $\lambda$ to accelerate convergence and stabilize constraint satisfaction.
  • Figure 3: Evolution of the model’s thinking patterns across RL iterations on AIME2024 and AIME2025, using DeepSeek-R1-Distill-Qwen-1.5B as the base model. The average response length (left) consistently decreases throughout training, indicating progressive compression of reasoning trajectories. Correspondingly, the frequencies of summary-, rethink-, and plan-related keywords (right) exhibit distinct dynamics: summary and rethink behaviors decline sharply before stabilizing with slight rebounds, while plan behaviors continuously diminish.