OPE: Overcoming Information Saturation in Parallel Thinking via Outline-Guided Path Exploration

TL;DR

This work identifies mutual information saturation among exploration paths as a key bottleneck in parallel thinking for large reasoning models under RLVR. It introduces Outline-Guided Parallel Exploration (OPE), which partitions the solution space with diverse outlines and uses an iterative RL strategy that alternates Outline Planning RL and Path Reasoning RL, starting from a cold-start. Empirically, OPE yields consistent improvements across diverse math benchmarks, especially on challenging tasks, and exhibits superior test-time scalability and reduced overthinking compared to naive path sampling. Analyses show that explicit outline-guided exploration increases solution diversity, shortens correct reasoning traces, and broadens the regions of the solution space explored, establishing a new baseline for efficient parallel reasoning.

Abstract

Parallel thinking has emerged as a new paradigm for large reasoning models (LRMs) in tackling complex problems. Recent methods leverage Reinforcement Learning (RL) to enhance parallel thinking, aiming to address the limitations in computational resources and effectiveness encountered with supervised fine-tuning. However, most existing studies primarily focus on optimizing the aggregation phase, with limited attention to the path exploration stage. In this paper, we theoretically analyze the optimization of parallel thinking under the Reinforcement Learning with Verifiable Rewards (RLVR) setting, and identify that the mutual information bottleneck among exploration paths fundamentally restricts overall performance. To address this, we propose Outline-Guided Path Exploration (OPE), which explicitly partitions the solution space by generating diverse reasoning outlines prior to parallel path reasoning, thereby reducing information redundancy and improving the diversity of information captured across exploration paths. We implement OPE with an iterative RL strategy that optimizes outline planning and outline-guided reasoning independently. Extensive experiments across multiple challenging mathematical benchmarks demonstrate that OPE effectively improves reasoning performance in different aggregation strategies, enabling LRMs to more reliably discover correct solutions.

Figures (5)

• Figure 1: Overview of OPE framework.(Left) Naive parallel thinking samples reasoning paths independently. Due to mode collapse, these paths often exhibit high redundancy and tend to converge on the same incorrect answer. (Middle) OPE mitigates this by explicitly generating diverse outlines to partition the solution space into distinct directions (four different strategies). This structured exploration maximizes the coverage of potential solutions, enabling the model to successfully locate the correct reasoning trajectory. (Right) The OPE training pipeline consists of a Cold Start phase using synthesized data, followed by a novel Iterative RL strategy.
• Figure 2: Comparisons of Pass@k curves and Maj@k curves on the HMMT-25 benchmark.
• Figure 3: Comparison between continued Outline Planning RL (from 70 steps) and Path Reasoning RL, evaluated using average accuracy across all benchmarks with Random aggregation.
• Figure 4: Pass@k scaling comparisons.
• Figure 5: Comparison of the performance curves of Naive and OPE approaches across different datasets as training steps progress.