TwT: Thinking without Tokens by Habitual Reasoning Distillation with Multi-Teachers' Guidance

TL;DR

TwT tackles the high inference cost of reasoning in LLMs by introducing a two-stage, unsupervised distillation framework that combines Dual-Criteria Rejection Sampling (DCRS) with Habitual Reasoning Distillation (HaRD). DCRS curates high-quality, diverse distillation data from multiple teachers, while HaRD progressively internalizes reasoning into the student through three stages: full reasoning, compressed reasoning, and reasoning-free distillation, shifting computation from inference to training. Empirically, TwT achieves up to a 13.6% accuracy improvement with substantially fewer output tokens across benchmarks, demonstrating robust performance gains and near-token-free inference on complex tasks like MetaMath. The approach is practical for deploying cost-efficient, reasoning-enabled LLMs without labeled data, benefiting search, API consumption, and first-pass AI context tasks.

Abstract

Large Language Models (LLMs) have made significant strides in problem-solving by incorporating reasoning processes. However, this enhanced reasoning capability results in an increased number of output tokens during inference, leading to higher computational costs. To address this challenge, we propose TwT (Thinking without Tokens), a method that reduces inference-time costs through habitual reasoning distillation with multi-teachers' guidance, while maintaining high performance. Our approach introduces a Habitual Reasoning Distillation method, which internalizes explicit reasoning into the model's habitual behavior through a Teacher-Guided compression strategy inspired by human cognition. Additionally, we propose Dual-Criteria Rejection Sampling (DCRS), a technique that generates a high-quality and diverse distillation dataset using multiple teacher models, making our method suitable for unsupervised scenarios. Experimental results demonstrate that TwT effectively reduces inference costs while preserving superior performance, achieving up to a 13.6% improvement in accuracy with fewer output tokens compared to other distillation methods, offering a highly practical solution for efficient LLM deployment.

Figures (12)

• Figure 1: Overview of the Proposed Method. The upper part of the figure illustrates the background problem: while generating more reasoning steps improves the performance of LLMs, it also leads to significantly higher computational costs. To mitigate this, we propose targeted strategies, shown in the lower part of the figure. Our approach reduces the cost per token by distilling knowledge from large models into smaller ones and minimizes the total number of tokens by gradually shrinking intermediate reasoning paths.
• Figure 2: Method Framework. Our proposed TwT (Thinking without Tokens) framework consists of two stages: Dual-Criteria Rejection Sampling (DCRS) and Habitual Reasoning Distillation (HaRD). In the first stage, DCRS selects a high-quality and diverse reasoning distillation dataset generated by multiple teacher LLMs (e.g., T1, T2, T3). In the second stage, HaRD progressively internalizes reasoning ability into the student model through a three-stage distillation process.
• Figure 3: Dual-Criteria Rejection Sampling Architecture. Our proposed DCRS method comprises two stages: Quality Selection and Diversity Selection. The first stage filters samples using confidence scores, while the second stage enhances diversity by selecting samples based on similarity. This approach ensures a high-quality and diverse distillation dataset, enabling our method to effectively adapt to unsupervised scenarios.
• Figure 4: Prompt for Teachers to Generate Pseudo-Labels on the MBPP Dataset.
• Figure 5: An example query of the MBPP dataset, showing the input programming task and its corresponding test case.