Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Making Small Language Models Efficient Reasoners: Intervention, Supervision, Reinforcement

Xuechen Zhang, Zijian Huang, Chenshun Ni, Ziyang Xiong, Jiasi Chen, Samet Oymak

TL;DR

The paper tackles the inefficiency of reasoning in small language models by showing that supervised fine-tuning often yields verbose, repetitive traces due to poor stopping control. It introduces two interventions: a training-free post-hoc Temperature Scaling (TS) to bias early termination and TLDR, a length-penalized reinforcement learning method based on GRPO that enables multi-level, user-guided trace lengths. Across four reasoning benchmarks, TS delivers a finer stopping-time control and improves efficiency over budget-forcing, while TLDR reduces token usage by about 50% with minimal accuracy loss and provides flexible length control through prompts. The work highlights stopping-time control as a key factor in efficient small-model reasoning and offers practical algorithmic recipes for token-efficient reasoning in constrained settings.

Abstract

Recent research enhances language model reasoning by scaling test-time compute via longer chain-of-thought traces. This often improves accuracy but also introduces redundancy and high computational cost, especially for small language models distilled with supervised fine-tuning (SFT). In this work, we propose new algorithms to improve token-efficient reasoning with small-scale models by effectively trading off accuracy and computation. We first show that the post-SFT model fails to determine the optimal stopping point of the reasoning process, resulting in verbose and repetitive outputs. Verbosity also significantly varies across wrong vs correct responses. To address these issues, we propose two solutions: (1) Temperature scaling (TS) to control the stopping point for the thinking phase and thereby trace length, and (2) TLDR: a length-regularized reinforcement learning method based on GRPO that facilitates multi-level trace length control (e.g. short, medium, long reasoning). Experiments on four reasoning benchmarks, MATH500, AMC, AIME24 and OlympiadBench, demonstrate that TS is highly effective compared to s1's budget forcing approach and TLDR significantly improves token efficiency by about 50% with minimal to no accuracy loss over the SFT baseline. Moreover, TLDR also facilitates flexible control over the response length, offering a practical and effective solution for token-efficient reasoning in small models. Ultimately, our work reveals the importance of stopping time control, highlights shortcomings of pure SFT, and provides effective algorithmic recipes.

Making Small Language Models Efficient Reasoners: Intervention, Supervision, Reinforcement

TL;DR

The paper tackles the inefficiency of reasoning in small language models by showing that supervised fine-tuning often yields verbose, repetitive traces due to poor stopping control. It introduces two interventions: a training-free post-hoc Temperature Scaling (TS) to bias early termination and TLDR, a length-penalized reinforcement learning method based on GRPO that enables multi-level, user-guided trace lengths. Across four reasoning benchmarks, TS delivers a finer stopping-time control and improves efficiency over budget-forcing, while TLDR reduces token usage by about 50% with minimal accuracy loss and provides flexible length control through prompts. The work highlights stopping-time control as a key factor in efficient small-model reasoning and offers practical algorithmic recipes for token-efficient reasoning in constrained settings.

Abstract

Recent research enhances language model reasoning by scaling test-time compute via longer chain-of-thought traces. This often improves accuracy but also introduces redundancy and high computational cost, especially for small language models distilled with supervised fine-tuning (SFT). In this work, we propose new algorithms to improve token-efficient reasoning with small-scale models by effectively trading off accuracy and computation. We first show that the post-SFT model fails to determine the optimal stopping point of the reasoning process, resulting in verbose and repetitive outputs. Verbosity also significantly varies across wrong vs correct responses. To address these issues, we propose two solutions: (1) Temperature scaling (TS) to control the stopping point for the thinking phase and thereby trace length, and (2) TLDR: a length-regularized reinforcement learning method based on GRPO that facilitates multi-level trace length control (e.g. short, medium, long reasoning). Experiments on four reasoning benchmarks, MATH500, AMC, AIME24 and OlympiadBench, demonstrate that TS is highly effective compared to s1's budget forcing approach and TLDR significantly improves token efficiency by about 50% with minimal to no accuracy loss over the SFT baseline. Moreover, TLDR also facilitates flexible control over the response length, offering a practical and effective solution for token-efficient reasoning in small models. Ultimately, our work reveals the importance of stopping time control, highlights shortcomings of pure SFT, and provides effective algorithmic recipes.
Paper Structure (21 sections, 1 equation, 16 figures, 7 tables)

This paper contains 21 sections, 1 equation, 16 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Observations of Supervised Finetuning and Reinforcement Learning Regarding Response Length
  4. Observation 1: Supervised fine-tuning introduces redundancy
  5. Observation 2: Longer traces for FT causes redundant responses for small models
  6. Observation 3: Controlling context length trades off performance for efficiency
  7. Observation 4: Test-time compute cannot precisely control response length
  8. Observation 5: SFT introduces redundancy due to equal treatment of end token, while RL-based models can learn when to stop
  9. Proposed Method: Post-hoc Temperature Scaling
  10. Proposed Method: Reinforcement Learning with Length Penalty
  11. Experiments
  12. Sweet Spot through Reinforcement Learning + Length Penalty
  13. Length Control through Reinforcement Learning + Multi-Level Length Penalty
  14. Conclusion
  15. Model performance
  16. ...and 6 more sections

Figures (16)

  • Figure 1: The average performances across four reasoning benchmarks for 7B models. Our RL-based length-control method TLDR enhances token efficiency by over 50% compared to SFT. Our temperature scaling (TS) method outperforms other test-time intervention techniques that avoid the need for training, such as Budget Forcing (BF) muennighoff2025s1 and Prompting. The detailed explanation of baselines and our methods are provided in Sec. \ref{['sec:method']}.
  • Figure 2: We explore training strategies for small language models (SLM), focusing on the effects on reasoning capability and test-time compute efficiency. RL improves efficiency, but when used alone, can compromise reasoning performance. While SFT enhances reasoning, it often leads to redundant outputs due to the lack of deliberate stopping point control. Test-time strategies offer limited control over output length and fail to improve performance with increased length consistently. To address these challenges, we propose TLDR, a method that combines SFT with RL to achieve both strong reasoning and token-efficient generation.
  • Figure 3: SFT models have longer and more repetitive answers when wrong, especially for small models fine-tuned under supervision by long traces.
  • Figure 4: Example of wrong and lengthy response, generated by the Deepseek-R1-Distill-Qwen-1.5B model for a Math500 question. The response repeatedly cycles through the paragraph shown in the blue block until it reaches the maximum context length, without providing a final answer.
  • Figure 5: Test time compute strategies can trade off performance for efficiency, but cannot precisely control response length, in part due to many tokens spent on the final solution.
  • ...and 11 more figures