CtrlCoT: Dual-Granularity Chain-of-Thought Compression for Controllable Reasoning

TL;DR

CtrlCoT introduces a dual-granularity approach to CoT compression that blends semantic-level abstraction with token-level pruning to achieve substantial token savings without sacrificing reasoning accuracy. The framework comprises Hierarchical Reasoning Abstraction (HRA) to generate multi-level semantic traces, Logic-Preserving Distillation (LPD) to retain critical logical cues during pruning, and Distribution-Alignment Generation (DAG) to produce distribution-matched, fluent traces. A Budget-Controlled Reasoner (BCR) and Budget-Free Reasoner (BFR) are trained to deliver controllable or automatic concise reasoning under user-specified or automatically determined budgets. Empirical results on GSM8K and MATH-500 across multiple model scales show that CtrlCoT delivers superior accuracy–token efficiency trade-offs, achieving notable improvements such as a $+7.6$ percentage-point accuracy gain with $30.7\%$ fewer tokens on MATH-500 with Qwen2.5-7B-Instruct and $55.7\%$ token reductions with negligible accuracy loss on GSM8K with Qwen2.5-14B-Instruct.

Abstract

Chain-of-thought (CoT) prompting improves LLM reasoning but incurs high latency and memory cost due to verbose traces, motivating CoT compression with preserved correctness. Existing methods either shorten CoTs at the semantic level, which is often conservative, or prune tokens aggressively, which can miss task-critical cues and degrade accuracy. Moreover, combining the two is non-trivial due to sequential dependency, task-agnostic pruning, and distribution mismatch. We propose \textbf{CtrlCoT}, a dual-granularity CoT compression framework that harmonizes semantic abstraction and token-level pruning through three components: Hierarchical Reasoning Abstraction produces CoTs at multiple semantic granularities; Logic-Preserving Distillation trains a logic-aware pruner to retain indispensable reasoning cues (e.g., numbers and operators) across pruning ratios; and Distribution-Alignment Generation aligns compressed traces with fluent inference-time reasoning styles to avoid fragmentation. On MATH-500 with Qwen2.5-7B-Instruct, CtrlCoT uses 30.7\% fewer tokens while achieving 7.6 percentage points higher than the strongest baseline, demonstrating more efficient and reliable reasoning. Our code will be publicly available at https://github.com/fanzhenxuan/Ctrl-CoT.

Figures (9)

• Figure 1: Accuracy versus CoT length on MATH-500 under different token budgets for Qwen2.5-7B/14B-Instruct, where our method consistently achieves higher accuracy at comparable CoT lengths.
• Figure 2: Framework of CtrlCoT. In the training stage, HRA first performs semantic-level compression, followed by token-level compression with LPD and DAG; the resulting CoTs are then aggregated for data pooling and model training. In the inference stage, the LLM performs efficient budget-conditioned reasoning given a user-specified token budget.
• Figure 3: The process of constructing a Budget-Free Reasoner. By training on the shortest correct CoTs, BFR enables efficient automatic reasoning.
• Figure 4: Accuracy versus CoT length on (a) GSM8K and (b) MATH-500, comparing our method with TokenSkip and Truncation under different compression strengths.
• Figure 5: (a) The impact of the minimum compression ratio when constructing token-level compressed CoTs. (b) Comparison between BFR and TokenSkip on MATH-500 for Qwen2.5-7B and Qwen2.5-14B. (c) CoT length of the original model versus the budget-free model (BFR) on MATH-500 across difficulty levels.