OPERA: A Reinforcement Learning--Enhanced Orchestrated Planner-Executor Architecture for Reasoning-Oriented Multi-Hop Retrieval

TL;DR

OPERA tackles the core challenges of reasoning-oriented multi-hop retrieval by decoupling strategic planning from tactical execution through a Goal Planning Module and a Reason-Execute Module, guided by a Trajectory Memory Component for interpretability. The framework introduces MAPGRPO, a staged, heterogeneous-reward extension of Group Relative Policy Optimization, enabling specialized agents (Plan, Analysis-Answer, Rewrite) to co-operate effectively. The approach yields strong improvements across HotpotQA, 2WikiMultiHopQA, and MuSiQue, with architectural ablations underscoring the critical role of planning and rewriting components. The work advances practical multi-hop reasoning in RAG by combining architectural design with information-theoretic, trajectory-aware training, offering robust performance and interpretability for complex retrieval tasks.

Abstract

Recent advances in large language models (LLMs) and dense retrievers have driven significant progress in retrieval-augmented generation (RAG). However, existing approaches face significant challenges in complex reasoning-oriented multi-hop retrieval tasks: 1) Ineffective reasoning-oriented planning: Prior methods struggle to generate robust multi-step plans for complex queries, as rule-based decomposers perform poorly on out-of-template questions. 2) Suboptimal reasoning-driven retrieval: Related methods employ limited query reformulation, leading to iterative retrieval loops that often fail to locate golden documents. 3) Insufficient reasoning-guided filtering: Prevailing methods lack the fine-grained reasoning to effectively filter salient information from noisy results, hindering utilization of retrieved knowledge. Fundamentally, these limitations all stem from the weak coupling between retrieval and reasoning in current RAG architectures. We introduce the Orchestrated Planner-Executor Reasoning Architecture (OPERA), a novel reasoning-driven retrieval framework. OPERA's Goal Planning Module (GPM) decomposes questions into sub-goals, which are executed by a Reason-Execute Module (REM) with specialized components for precise reasoning and effective retrieval. To train OPERA, we propose Multi-Agents Progressive Group Relative Policy Optimization (MAPGRPO), a novel variant of GRPO. Experiments on complex multi-hop benchmarks show OPERA's superior performance, validating both the MAPGRPO method and OPERA's design.

Figures (11)

• Figure 1: Overview of OPERA's MAPGRPO training framework and performance comparison with traditional RAG.
• Figure 2: Overview of OPERA architecture showing the Goal Planning Module (GPM) with Plan Agent for strategic decomposition, and the Reason-Execute Module (REM) with Analysis-Answer and Rewrite Agents for adaptive execution. The Trajectory Memory Component (TMC) will record all things.
• Figure 3: OPERA's runtime dynamics. (Left) Agent call intensity and question completion rate over processing steps. (Right) Attention visualization across query token types and processing stages.
• Figure 4: Training dynamics for MAPGRPO. (Top) Average reward curves show stable learning. (Bottom) Plots confirm controlled policy divergence (left) and significant gradient variance reduction from expert injection (right).
• Figure 5: Reward distribution evolution during MAPGRPO training for Plan and Analysis-Answer agents.