Correct, Concise and Complete: Multi-stage Training For Adaptive Reasoning

TL;DR

This work tackles the inefficiency of lengthy chain-of-thought in large language models by proposing a multi-stage training pipeline that first uses supervised fine-tuning with concise reasoning traces and then optimizes via reinforcement learning with an adaptive length penalty. It introduces two SFT data strategies—rejection sampling and trace reformats—and a verifiable RL reward that penalizes tokens generated after the first correct answer, achieving significant reductions in response length with only modest accuracy loss across diverse reasoning tasks and model sizes. Through comprehensive evaluation against baselines and state-of-the-art efficient reasoning methods, the approach improves the Overthinking-Adjusted Accuracy curve (AUC_OAA) and demonstrates practical gains in inference efficiency. The results offer insights into how reasoning traces adapt to varying task domains and difficulty, suggesting broad applicability for cost-effective reasoning in real-world settings.

Abstract

The reasoning capabilities of large language models (LLMs) have improved substantially through increased test-time computation, typically in the form of intermediate tokens known as chain-of-thought (CoT). However, CoT often becomes unnecessarily long, increasing computation cost without actual accuracy gains or sometimes even degrading performance, a phenomenon known as ``overthinking''. We propose a multi-stage efficient reasoning method that combines supervised fine-tuning -- via rejection sampling or reasoning trace reformatting -- with reinforcement learning using an adaptive length penalty. We introduce a lightweight reward function that penalizes tokens generated after the first correct answer but encouraging self-verification only when beneficial. We conduct a holistic evaluation across seven diverse reasoning tasks, analyzing the accuracy-response length trade-off. Our approach reduces response length by an average of 28\% for 8B models and 40\% for 32B models, while incurring only minor performance drops of 1.6 and 2.5 points, respectively. Despite its conceptual simplicity, it achieves a superior trade-off compared to more complex state-of-the-art efficient reasoning methods, scoring 76.6, in terms of the area under the Overthinking-Adjusted Accuracy curve ($\text{AUC}_{\text{OAA}}$) -- 5 points above the base model and 2.5 points above the second-best approach.

Figures (6)

• Figure 1: Overthinking-Adjusted Accuracy (OAA) aggarwal2025optimalthinkingbench as a function of the response length threshold on MATH-500 for Qwen3-8B. Our approach achieves similar accuracy with fewer tokens, leading to a larger area under the curve.
• Figure 2: Average accuracy versus number of tokens for each method using Qwen3-8B. Points in the green region are dominated by Adaptive-Answer or Format-Adaptive-Answer, while points in the orange region dominate them (higher accuracy, fewer tokens).
• Figure 3: Response length distributions of some representative efficient reasoning methods applied to Qwen3-8B. We separate the correct and incorrect responses.
• Figure 4: Distributions of the number of correct answers in the traces of some representative efficient reasoning methods applied to Qwen3-8B for MATH-500, AIME 24 and AIME 25.
• Figure 5: Accuracy, response length, and count of intermediate correct steps across difficulty levels on MATH-500.