ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoning

TL;DR

This work tackles the inefficiency of reasoning traces in large language models by enabling autonomous discovery of optimal inference length. It introduces Sample Optimal Length (SOL), a self-supervised reward based on the shortest correct reasoning among multiple samples, and optimizes with Group Relative Policy Optimization to encourage concise yet accurate CoTs. Empirically, ShorterBetter achieves substantial output-length reductions (roughly 50-80%) on 1.5B/7B base models while preserving or improving accuracy across in-domain math tasks and maintaining competitive performance on out-of-domain benchmarks. The paper also introduces reasoning-structure analyses and ablations showing that SOL improves the quality and efficiency of reasoning traces, with robustness validated by human and automated evaluations.

Abstract

Recent models such as OpenAI o1 and DeepSeek-R1 have demonstrated strong performance on reasoning-intensive tasks by generating extended Chain-of-Thought (CoT) traces. While longer reasoning helps with thorough exploration of solution paths for complex problems, it also often leads to inefficient and redundant outputs--a phenomenon commonly described as overthinking. In this paper, we propose ShorterBetter, a simple yet effective reinforcement learning method that enables reasoning models to learn their own optimal CoT lengths without manual supervision. We define the Sample Optimal Length (SOL) as the length of the shortest correct response among multiple generations, which serves as a dynamic reward signal to guide the model toward efficient reasoning. Applied to DeepSeek-Distill-Qwen-1.5B/7B as base models, ShorterBetter achieves 50%-80% reduction in output lengths in both in-domain and out-of-domain reasoning tasks while maintaining accuracy. Our reasoning trace analysis shows that ShorterBetter refines the structure of the reasoning traces by reducing unnecessary repetition, excessive self-verification, and over-exploration of alternatives.

Figures (9)

• Figure 1: ShorterBetter exhibits strong length reduction performance (1.5B models). We report accuracy and output length of five sets of models on various tasks. Models include DeepSeek-R1-Distill-Qwen-1.5B, Qwen2.5-1.5B-Instruct (non-reasoning model), ShorterBetter-1.5B ($\alpha$ =1 and $\alpha$ =2) and Training Efficent-1.5B arora2025training. Our method achieves significant length reduction compared to all baseline reasoning models while maintaining accuracy.
• Figure 2: Flowchart for the ShorterBetter pipeline. We detail the reward design in \ref{['subsec:sol']} and the GRPO algorithm in \ref{['subsec:grpo']}. Reward values in display are rescaled for better illustration.
• Figure 3: Reasoning trace structure analysis. Left: Percentage of remaining output after first appearance of the correct answer; Right: Token-level distribution of reasoning categories on the MATH benchmark.
• Figure 4: Sample optimal length inherently combines correctness and conciseness. Remove the correctness rewarding term in the reward function but keep the entire design of SOL, i.e., set $\alpha=0$.
• Figure 5: Naively adopting the shortest response as target length leads to training collapses. Ablation results show that replacing or partially replacing SOL with the shortest response length, with or without correctness reward, leads to training collapse within 100 steps.