Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

D$^2$Plan: Dual-Agent Dynamic Global Planning for Complex Retrieval-Augmented Reasoning

Kangcheng Luo, Tinglang Wu, Yansong Feng

TL;DR

This work tackles failures in retrieval-augmented reasoning where long contexts cause query drift and distractor-driven errors. It introduces D^2Plan, a dual-agent framework with a Reasoner and Purifier that perform dynamic global planning and evidence purification, trained via a two-stage pipeline: SFT cold-start and plan-oriented RL (SPlanRL). Plan-oriented rewards $R_p$, $R_a$, and $R_{ans}$, plus a format constraint reward, guide initial plan construction, adaptation, and revisions, yielding coherent multi-hop reasoning and resilience to irrelevant evidence. Empirically, D^2Plan achieves consistent improvements across six QA benchmarks and scales to 7B parameter families, with notable gains on harder tasks and improved efficiency through purification, underscoring the value of explicit planning and retrieval-quality control in complex open-domain reasoning.

Abstract

Recent search-augmented LLMs trained with reinforcement learning (RL) can interleave searching and reasoning for multi-hop reasoning tasks. However, they face two critical failure modes as the accumulating context becomes flooded with both crucial evidence and irrelevant information: (1) ineffective search chain construction that produces incorrect queries or omits retrieval of critical information, and (2) reasoning hijacking by peripheral evidence that causes models to misidentify distractors as valid evidence. To address these challenges, we propose **D$^2$Plan**, a **D**ual-agent **D**ynamic global **Plan**ning paradigm for complex retrieval-augmented reasoning. **D$^2$Plan** operates through the collaboration of a *Reasoner* and a *Purifier*: the *Reasoner* constructs explicit global plans during reasoning and dynamically adapts them based on retrieval feedback; the *Purifier* assesses retrieval relevance and condenses key information for the *Reasoner*. We further introduce a two-stage training framework consisting of supervised fine-tuning (SFT) cold-start on synthesized trajectories and RL with plan-oriented rewards to teach LLMs to master the **D$^2$Plan** paradigm. Extensive experiments demonstrate that **D$^2$Plan** enables more coherent multi-step reasoning and stronger resilience to irrelevant information, thereby achieving superior performance on challenging QA benchmarks.

D$^2$Plan: Dual-Agent Dynamic Global Planning for Complex Retrieval-Augmented Reasoning

TL;DR

This work tackles failures in retrieval-augmented reasoning where long contexts cause query drift and distractor-driven errors. It introduces D^2Plan, a dual-agent framework with a Reasoner and Purifier that perform dynamic global planning and evidence purification, trained via a two-stage pipeline: SFT cold-start and plan-oriented RL (SPlanRL). Plan-oriented rewards , , and , plus a format constraint reward, guide initial plan construction, adaptation, and revisions, yielding coherent multi-hop reasoning and resilience to irrelevant evidence. Empirically, D^2Plan achieves consistent improvements across six QA benchmarks and scales to 7B parameter families, with notable gains on harder tasks and improved efficiency through purification, underscoring the value of explicit planning and retrieval-quality control in complex open-domain reasoning.

Abstract

Recent search-augmented LLMs trained with reinforcement learning (RL) can interleave searching and reasoning for multi-hop reasoning tasks. However, they face two critical failure modes as the accumulating context becomes flooded with both crucial evidence and irrelevant information: (1) ineffective search chain construction that produces incorrect queries or omits retrieval of critical information, and (2) reasoning hijacking by peripheral evidence that causes models to misidentify distractors as valid evidence. To address these challenges, we propose **DPlan**, a **D**ual-agent **D**ynamic global **Plan**ning paradigm for complex retrieval-augmented reasoning. **DPlan** operates through the collaboration of a *Reasoner* and a *Purifier*: the *Reasoner* constructs explicit global plans during reasoning and dynamically adapts them based on retrieval feedback; the *Purifier* assesses retrieval relevance and condenses key information for the *Reasoner*. We further introduce a two-stage training framework consisting of supervised fine-tuning (SFT) cold-start on synthesized trajectories and RL with plan-oriented rewards to teach LLMs to master the **DPlan** paradigm. Extensive experiments demonstrate that **DPlan** enables more coherent multi-step reasoning and stronger resilience to irrelevant information, thereby achieving superior performance on challenging QA benchmarks.
Paper Structure (37 sections, 1 equation, 4 figures, 27 tables)

This paper contains 37 sections, 1 equation, 4 figures, 27 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Overview of D2Plan
  4. SFT Cold-Start
  5. Synthesizing Trajectories
  6. Constructing SFT Data
  7. SPlanRL
  8. Experimental Setup
  9. Results and Analysis
  10. Main Results (RQ1)
  11. Ablation Study (RQ1)
  12. Retrieval Quality (RQ2)
  13. Effect of SPlanRL (RQ3)
  14. Plan Adaptation Reward Steers Correct Plan Revisions.
  15. Effect of Dual-Agent Architecture (RQ4)
  16. ...and 22 more sections

Figures (4)

  • Figure 1: Illustrative examples of 2 typical error types.
  • Figure 2: An illustrative example of our D$^2$Plan paradigm resolving a multi-hop reasoning question, where the correct number of sub-questions in the initial plan and effective plan adaptation are rewarded in the RL phase.
  • Figure 3: Overview of the dual-agent architecture.
  • Figure 4: Error type distribution on FRAMES and MuSiQue. Percentages are calculated relative to the total number of questions, including correct predictions.