A Survey on Parallel Reasoning

TL;DR

This survey analyzes parallel reasoning (PR) as an inference paradigm that broadens problem-solving by generating and aggregating multiple concurrent reasoning paths, addressing fragility in sequential CoT. It formalizes PR as a three-stage pipeline with decomposition, parallel processing, and aggregation, and categorizes methods into non-interactive, interactive, and efficiency-focused families, including self-consistency, ranking-based selection, Tree/Graph-of-Thoughts, and multi-agent collaboration. The paper surveys practical techniques (e.g., BoN, verifiers, adaptive termination, parallel decoding, speculative decoding) and highlights applications in grand challenges, reliability, workflow acceleration, and creativity, while outlining core challenges such as Pass@k limitations and the need for end-to-end optimization. It concludes that PR offers significant robustness and efficiency benefits, but achieving scalable, end-to-end training and multimodal integration remains an open, active area with substantial practical impact. The work provides a comprehensive roadmap for researchers and practitioners aiming to leverage breadth-first reasoning in real-world AI systems, with a focus on formalism, architectures, and optimization strategies such as latency-aware budgets and external tool coordination.

Abstract

With the increasing capabilities of Large Language Models (LLMs), parallel reasoning has emerged as a new inference paradigm that enhances reasoning robustness by concurrently exploring multiple lines of thought before converging on a final answer. It has become a significant trend to explore parallel reasoning to overcome the fragility of standard sequential methods and improve practical performance. In this paper, we aim to survey and summarize the progress and challenges of parallel reasoning. We first present a formal definition of parallel reasoning and clarify its distinction from related concepts like Chain-of-Thought. Then, we organize and discuss advanced techniques based on a novel taxonomy, including non-interactive reasoning, interactive reasoning, and efficiency-focused decoding strategies. Additionally, we explore various application scenarios, such as solving complex problems and enhancing the reliability of LLM outputs.Finally, we highlight the core challenges of parallel reasoning and suggest potential directions for future research. We hope that our work can provide a useful roadmap for beginners and encourage more research on improving parallel reasoning methods. Related source can be avaliable in https://github.com/PPPP-kaqiu/Awesome-Parallel-Reasoning.

Figures (2)

• Figure 1: An overall framework for a recurrent parallel reasoning paradigm. The process begins with a decomposition stage, followed by parallel processing of sub-tasks, and concludes by aggregating the outputs into a single answer.
• Figure 2: Taxonomy of Parallel Reasoning