Upfront Chain-of-Thought: A Cooperative Framework for Chain-of-Thought Compression

TL;DR

UCoT introduces a cooperative compressor-executor framework that converts lengthy CoT into dense Upfront Thoughts produced by a small compressor and consumed by a large executor to generate concise reasoning and final answers. The two-stage training (Upfront Thought Generation and Utilization) pairs a distillation loss for UTs with a reward-driven, semantically aligned objective to preserve reasoning quality while reducing token output. Empirical results on GSM8K and MATH with open-source models show substantial reductions in CoT length (~50%) and competitive or improved accuracy relative to state-of-the-art CoT compression methods, with demonstrated robustness to ultra-long CoT scenarios. The approach generalizes across model scales and tasks, offering a practical path to more efficient yet still capable LLM reasoning in real-world deployments.

Abstract

Recent developments have enabled advanced reasoning in Large Language Models (LLMs) via long Chain-of-Thought (CoT), while long CoT suffers from high computational costs and significant latency losses owing to the autoregressive nature of generative LLMs. CoT compression aims to improve efficiency in the reasoning process by reducing output length. Previous works trade reasoning efficiency by either laborious discrete prompt designing or the construction of external compressed CoT datasets that sacrifice key reasoning details. In this work, we propose Upfront CoT (UCoT): an efficient reasoning framework with upfront thought embedding to automate CoT compression. UCoT is a cooperative workflow involving a small model (compressor) and a large model (executor). The first stage of UCoT trains compressor to generate upfront thought embeddings rich in reasoning information for the executor, avoiding the drawbacks of manually designed prompts. The second stage optimizes executor to utilize upfront thought embeddings to derive the correct answer with short reasoning, using a reward mechanism. Extensive experiments show that UCoT maintains the powerful reasoning ability of executor while significantly reducing the length of CoT. It is worth mentioning that when applying UCoT to the Qwen2.5-7B-Instruct model, the usage of tokens on GSM8K dataset is reduced by 50\%, while the performance is 3.08\% higher than that of the state-of-the-art (SOTA) method. The code and dataset are in supplementary material.

Figures (9)

• Figure 1: A comparison of reasoning accuracy and the number of tokens output by LLMs between CoT$^{\dagger}$, CoT$^{*}$ and UCoT in GSM8K 2021arXiv211014168C . Our UCoT better balance the accuracy-efficiency trade-offs.
• Figure 2: Overview of Upfront Chain-of-Thought. In Upfront Thought Generation stage (§\ref{['section:3.1']}) we train a compressor $\mathcal{M}_c$ for efficiently and automatically generating Upfront Thought. In Upfront Thought Utilization stage (§\ref{['section:3.2']}), the executor $\mathcal{M}_{e}$ learns to give correct responses with less reasoning length based on the UT. In More Efficient Inference satge (§\ref{['section:3.3']}), the compressor $\mathcal{M}_c$ and the executor $\mathcal{M}_{e}$ cooperate to achieve efficient reasoning.
• Figure 3: Performance comparison between original CoT and UTs of different lengths.
• Figure 4: A case study on GSM8K dataset. Output using Qwen2.5-7B-Instruct as the executor and Qwen2.5-1.5B as the compressor.
• Figure 5: A case study on Math dataset. Output using Qwen2.5-7B-Instruct as the executor and Qwen2.5-1.5B as the compressor.