Multi-Path Collaborative Reasoning via Reinforcement Learning

TL;DR

The paper addresses the deterministic nature of traditional Chain-of-Thought decoding by introducing M3PO, a multi-path collaborative reinforcement learning framework that runs parallel reasoning trajectories and lets them inform each other via a light-weight gating mechanism. By integrating cross-path insights during thinking steps and optimizing with a group-relative objective, M3PO achieves state-of-the-art performance on both knowledge-intensive and STEM reasoning benchmarks using compact models, while preserving efficiency and interpretability. Ablation studies confirm the importance of cross-path fusion and the need to balance intrinsic and peer-derived signals, with lambda and temperature acting as critical knobs. The work demonstrates that multi-path collaborative reasoning can robustly improve complex problem solving without adding parameter overhead, suggesting a practical path toward more reliable LLM reasoning systems.

Abstract

Chain-of-Thought (CoT) reasoning has significantly advanced the problem-solving capabilities of Large Language Models (LLMs), yet conventional CoT often exhibits internal determinism during decoding, limiting exploration of plausible alternatives. Recent methods attempt to address this by generating soft abstract tokens to enable reasoning in a continuous semantic space. However, we find that such approaches remain constrained by the greedy nature of autoregressive decoding, which fundamentally isolates the model from alternative reasoning possibilities. In this work, we propose Multi-Path Perception Policy Optimization (M3PO), a novel reinforcement learning framework that explicitly injects collective insights into the reasoning process. M3PO leverages parallel policy rollouts as naturally diverse reasoning sources and integrates cross-path interactions into policy updates through a lightweight collaborative mechanism. This design allows each trajectory to refine its reasoning with peer feedback, thereby cultivating more reliable multi-step reasoning patterns. Empirical results show that M3PO achieves state-of-the-art performance on both knowledge- and reasoning-intensive benchmarks. Models trained with M3PO maintain interpretability and inference efficiency, underscoring the promise of multi-path collaborative learning for robust reasoning.

Figures (22)

• Figure 1: An illustrative case of probability distribution in vanilla Soft Thinking method. Each step aligns closely with the top-1 token from the prior step, leading to rapid concentration along a single semantic path.
• Figure 2: Overview of the M3PO framework. For a given question, multiple trajectories are first generated in parallel (Rollout). During thinking, these paths dynamically interact via a novel collaborative mechanism to refine intermediate reasoning steps (Collaborative Learning). The optimized trajectories are then used for policy updates via group-relative advantage estimation, closing the loop between multi-path exploration and policy learning (Policy Optimization).
• Figure 3: Training performance across latent reasoning variants. M3PO not only converges more rapidly but also attains consistently higher final rewards, underscoring the efficacy and robustness of our hybrid reasoning paradigm in complex reasoning tasks.
• Figure 4: Sensitivity of the blending coefficient $\lambda$ in Equation (\ref{['eq:hybrid']}). Performance collapses when $\lambda \ge 0.5$, highlighting the critical balance between intrinsic reasoning direction and peer insights.
• Figure 5: Ablation study on the impact of the distribution-similarity fusion mechanism. M3PO achieves the highest rewards, highlighting the effectiveness of our warm cross-path fusion strategy and the value of peer insights.