Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropy
Zeju Li, Jianyuan Zhong, Ziyang Zheng, Xiangyu Wen, Zhijian Xu, Yingying Cheng, Fan Zhang, Qiang Xu
TL;DR
This work tackles the inefficiency of verbose Chain-of-Thought reasoning in large language models by introducing step entropy, a metric that quantifies the informational contribution of individual reasoning steps. By showing that many low-entropy steps are redundant, the authors demonstrate that up to 80% of such steps can be pruned with minimal impact on accuracy, yielding substantial token reductions across multiple model families and benchmarks. They further enable autonomous compressed CoT generation through a two-stage training pipeline combining Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO), guiding models to insert [SKIP] tokens strategically. The approach delivers strong, model-agnostic improvements in inference efficiency while preserving reasoning quality, with detailed experiments on GSM8k, Math500, AIME, and MMLU that validate generalizability and scalability. A public code release accompanies the work, highlighting its practical relevance for scalable and transparent LLM deployments.
Abstract
Large Language Models (LLMs) using Chain-of-Thought (CoT) prompting excel at complex reasoning but generate verbose thought processes with considerable redundancy, leading to increased inference costs and reduced efficiency. We introduce a novel CoT compression framework based on step entropy, a metric that quantifies \emph{the informational contribution of individual reasoning steps} to identify redundancy. Through theoretical analysis and extensive empirical validation on mathematical reasoning benchmarks, we demonstrate that steps with low entropy are indeed highly redundant. Our experiments reveal that an astonishing 80\% of low-entropy intermediate steps can be pruned with minor degradation in the final answer accuracy across DeepSeek-R1-7B, 14B and Qwen3-8B. This finding sharply contrasts with random or high-entropy pruning, which severely impairs reasoning performance. Building on this, we propose a novel two-stage training strategy combining Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO) reinforcement learning. This approach enables LLMs to autonomously learn to generate compressed COTs during inference by strategically incorporating [SKIP] tokens. Our method significantly improves LLM inference efficiency while preserving accuracy, paving the way for more scalable LLM deployments and a better understanding of their internal reasoning. The code and data are released in https://github.com/staymylove/COT_Compresstion_via_Step_entropy.