ConMax: Confidence-Maximizing Compression for Efficient Chain-of-Thought Reasoning

TL;DR

ConMax addresses the inefficiency of verbose chain-of-thought reasoning in large reasoning models by learning a compression policy through reinforcement learning guided by dual confidence rewards derived from a frozen auxiliary LLM. It achieves substantial CoT length reduction (up to 43%) with minimal accuracy loss (as low as 0.7%), demonstrated across five reasoning benchmarks on the Qwen-2.5 family. The method does not require ground-truth compressed traces and yields data-efficient, high-quality training data for downstream training, outperforming heuristic prompt-based compression. By balancing predictive fidelity and reasoning coherence, ConMax offers a practical path to efficient, scalable reasoning in LRMs.

Abstract

Recent breakthroughs in Large Reasoning Models (LRMs) have demonstrated that extensive Chain-of-Thought (CoT) generation is critical for enabling intricate cognitive behaviors, such as self-verification and backtracking, to solve complex tasks. However, this capability often leads to ``overthinking'', where models generate redundant reasoning paths that inflate computational costs without improving accuracy. While Supervised Fine-Tuning (SFT) on reasoning traces is a standard paradigm for the 'cold start' phase, applying existing compression techniques to these traces often compromises logical coherence or incurs prohibitive sampling costs. In this paper, we introduce ConMax (Confidence-Maximizing Compression), a novel reinforcement learning framework designed to automatically compress reasoning traces while preserving essential reasoning patterns. ConMax formulates compression as a reward-driven optimization problem, training a policy to prune redundancy by maximizing a weighted combination of answer confidence for predictive fidelity and thinking confidence for reasoning validity through a frozen auxiliary LRM. Extensive experiments across five reasoning datasets demonstrate that ConMax achieves a superior efficiency-performance trade-off. Specifically, it reduces inference length by 43% over strong baselines at the cost of a mere 0.7% dip in accuracy, proving its effectiveness in generating high-quality, efficient training data for LRMs.

Figures (4)

• Figure 1: Comparison of reasoning traces generated by Qwen2.5-14B models fine-tuned on the original verbose data (top) and our ConMax-compressed data (bottom). Unlike the baseline, the model trained with ConMax generates a streamlined and logically coherent chain-of-thought, retaining essential reasoning patterns with significantly reduced token usage.
• Figure 2: Overview of ConMax. The framework uses reinforcement learning to train a policy $\pi_\theta$ for compressing originally verbose reasoning chains. A composite reward function guides the training by balancing Answer Confidence and Thinking Confidence during compression.
• Figure 3: Distribution of generated token lengths for correctly predicted cases, comparing ConMax against the Original Baseline. The ConMax distribution exhibits a stronger peak at shorter token lengths, indicating more concise generation compared to the baseline.
• Figure 4: The system prompt to instruct the policy to compress verbose reasoning chains.