PRISMA: Reinforcement Learning Guided Two-Stage Policy Optimization in Multi-Agent Architecture for Open-Domain Multi-Hop Question Answering

TL;DR

PRISMA introduces a decoupled, reinforcement-learning guided framework for open-domain multi-hop QA, integrating Plan-Retrieve-Inspect-Solve-Memoize agents to improve evidence gathering and grounded reasoning. The training uses a two-stage GRPO with Observation-Aware Residual Policy Optimization, calibrating planners and solvers before training a trajectory-conditioned Inspector to detect and recover from failures. Empirical results across ten benchmarks show state-of-the-art performance among trained systems and robustness advantages over API models, with detailed ablations and efficiency analyses. The work demonstrates the value of reasoning-guided collaboration and targeted recovery for reliable real-world multi-hop QA, while acknowledging resource demands and remaining gaps to API-based models.

Abstract

Answering real-world open-domain multi-hop questions over massive corpora is a critical challenge in Retrieval-Augmented Generation (RAG) systems. Recent research employs reinforcement learning (RL) to end-to-end optimize the retrieval-augmented reasoning process, directly enhancing its capacity to resolve complex queries. However, reliable deployment is hindered by two obstacles. 1) Retrieval Collapse: iterative retrieval over large corpora fails to locate intermediate evidence containing bridge answers without reasoning-guided planning, causing downstream reasoning to collapse. 2) Learning Instability: end-to-end trajectory training suffers from weak credit assignment across reasoning chains and poor error localization across modules, causing overfitting to benchmark-specific heuristics that limit transferability and stability. To address these problems, we propose PRISMA, a decoupled RL-guided framework featuring a Plan-Retrieve-Inspect-Solve-Memoize architecture. PRISMA's strength lies in reasoning-guided collaboration: the Inspector provides reasoning-based feedback to refine the Planner's decomposition and fine-grained retrieval, while enforcing evidence-grounded reasoning in the Solver. We optimize individual agent capabilities via Two-Stage Group Relative Policy Optimization (GRPO). Stage I calibrates the Planner and Solver as specialized experts in planning and reasoning, while Stage II utilizes Observation-Aware Residual Policy Optimization (OARPO) to enhance the Inspector's ability to verify context and trigger targeted recovery. Experiments show that PRISMA achieves state-of-the-art performance on ten benchmarks and can be deployed efficiently in real-world scenarios.

Figures (6)

• Figure 1: Challenges examples in real-world open-domain multi-hop QA.
• Figure 2: PRISMA mirrors human researcher workflows for multi-hop QA.Left: Researchers tackle complex questions (e.g., comparing protein inhibitors from the 2021 Nobel-winning TRPV1 and 2025 Spike protein studies) by (1) Planning the problem; (2) Consulting materials; (3) Reasoning by analyzing evidence and checking relevance; (4) Saving key findings. Right:PRISMA replicates this with agents: Planner decomposes questions, Retriever gathers documents, Inspector validates relevance and reasoning, Solver synthesizes answers, and Memoize records solved subquestions. Targeted retries close the retrieval-reasoning loop when issues are detected.
• Figure 3: We adopt a two-stage GRPO pipeline. Stage I calibrates the Planner and Solver on $\mathcal{D}_P$ and $\mathcal{D}_S$. Stage II constructs $\mathcal{D}_I$ by running the frozen $\text{System}^{(1)}$ with a high-capacity oracle Inspector teacher $g_T$ to obtain audit/recovery labels, mining successful rollouts (EM$=1$), and training a trajectory-conditioned Inspector on $\mathcal{D}_I$ with reward $R_{\text{OARPO}}$ (Appendix \ref{['sec:stage2_data_app']}). This reduces a trace-level (data) residual by aligning to $g_T$, and a system-level (policy) residual by learning better audit-and-recovery actions on top of the frozen experts.
• Figure 4: Training reward curves for Planner, Solver and Inspector.
• Figure 5: Top: (a) PRISMA's sequential agent interaction flow for answering "When is the opening day of the league, that includes the team that played in the most games of the event that the MLB MVP award is handed out after?" Bottom: (b) End-to-end latency breakdown across 100 test cases.