Self-Compression of Chain-of-Thought via Multi-Agent Reinforcement Learning

TL;DR

SCMA reframes Chain-of-Thought compression as a multi-agent reinforcement learning problem, introducing three roles (Reasoning, Segmentation, Scoring) that jointly optimize an importance-weighted length penalty to prune redundancy without sacrificing essential reasoning. The approach uses a shared reward with group-relative optimization to align agents, with zero test-time overhead since only the Reasoning Agent is deployed. Empirical results across GSM8K, MATH500, AMC23, and AIME show substantial reductions in reasoning length (11.1%–39.0%) and accuracy gains (4.33%–10.02%), with strong generalization. The work reveals emergent, content-adaptive segmentation and scoring dynamics that yield concise yet robust long-chain reasoning while preserving critical logic.

Abstract

The inference overhead induced by redundant reasoning undermines the interactive experience and severely bottlenecks the deployment of Large Reasoning Models. Existing reinforcement learning (RL)-based solutions tackle this problem by coupling a length penalty with outcome-based rewards. This simplistic reward weighting struggles to reconcile brevity with accuracy, as enforcing brevity may compromise critical reasoning logic. In this work, we address this limitation by proposing a multi-agent RL framework that selectively penalizes redundant chunks, while preserving essential reasoning logic. Our framework, Self-Compression via MARL (SCMA), instantiates redundancy detection and evaluation through two specialized agents: \textbf{a Segmentation Agent} for decomposing the reasoning process into logical chunks, and \textbf{a Scoring Agent} for quantifying the significance of each chunk. The Segmentation and Scoring agents collaboratively define an importance-weighted length penalty during training, incentivizing \textbf{a Reasoning Agent} to prioritize essential logic without introducing inference overhead during deployment. Empirical evaluations across model scales demonstrate that SCMA reduces response length by 11.1\% to 39.0\% while boosting accuracy by 4.33\% to 10.02\%. Furthermore, ablation studies and qualitative analysis validate that the synergistic optimization within the MARL framework fosters emergent behaviors, yielding more powerful LRMs compared to vanilla RL paradigms.

Figures (5)

• Figure 1: Overview of SCMA Compared to general RL with length penalty. (Left) The general RL calculates rewards by penalizing the length of the thinking process directly. (Right) The SCMA employs an importance-weighted length penalty within a multi-agent system.
• Figure 2: Average accuracy and response length of Qwen3-4B trained with different $\alpha$
• Figure 3: Training curves of SCMA and RL+LP. During training, the RL+LP model suffers from training collapse, with response length dropping significantly, indicating a "No Think" pattern.
• Figure 4: Evolution of the Scoring and Segmentation Agents. (Left) Curves showing the Average Score (orange), Average Chunk Num (blue), Chunk Length Std (red), the Chunk Length Std if Segmentation Agents are not optimized, for shot w/o optimization (green) over training steps. (Right) Case of agents at Step 10 versus Step 40, including segmentation points and score annotations.
• Figure 5: Dynamic evolution of response length and length penalty during SCMA training. The solid blue line (left axis) denotes the mean response length, while the dashed grey line (right axis) represents the importance-weighted length penalty. The figure illustrates how the dynamic penalty modulates response length over training steps to ensure convergence and stability.