Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revision

TL;DR

Deep Research Agents are rarely reliable at multi-turn report revision, as current models can follow user feedback but frequently regress on previously covered content and citation quality. The authors present Mr Dre, a unified evaluation suite combining a three-dimension long-form report protocol with a human-verified feedback pipeline to study iterative revision. Across five DRAs and three feedback settings, inference-time fixes like prompt engineering or a reviser sub-agent improve certain signals but fail to guarantee preservation of content and citations, highlighting a need for foundational advances in training and scaffolding. Mr Dre thus provides a practical benchmark and a clear direction for building DRAs capable of reliable, iterative writing in research contexts.

Abstract

Existing benchmarks for Deep Research Agents (DRAs) treat report generation as a single-shot writing task, which fundamentally diverges from how human researchers iteratively draft and revise reports via self-reflection or peer feedback. Whether DRAs can reliably revise reports with user feedback remains unexplored. We introduce Mr Dre, an evaluation suite that establishes multi-turn report revision as a new evaluation axis for DRAs. Mr Dre consists of (1) a unified long-form report evaluation protocol spanning comprehensiveness, factuality, and presentation, and (2) a human-verified feedback simulation pipeline for multi-turn revision. Our analysis of five diverse DRAs reveals a critical limitation: while agents can address most user feedback, they also regress on 16-27% of previously covered content and citation quality. Over multiple revision turns, even the best-performing agents leave significant headroom, as they continue to disrupt content outside the feedback's scope and fail to preserve earlier edits. We further show that these issues are not easily resolvable through inference-time fixes such as prompt engineering and a dedicated sub-agent for report revision.

Figures (29)

• Figure 1: Illustrative example of multi-turn revision failure in Deep Research Agents. The revised report incorporates the user feedback but removes previously covered content that is outside the feedback's scope.
• Figure 2: Mr Dre Evaluation Suite. In multi-turn report revision, a DRA iteratively drafts and revises a report $r$ for question $q$ given user feedback $f$ (top left). Mr Dre provides a unified Deep Research report evaluation protocol (bottom left) along three dimensions: Comprehensiveness, Factuality, and Presentation. To evaluate multi-turn revision performance, Mr Dre provides a pipeline to simulate content, format, and self-reflection feedback (right).
• Figure 3: Results for extending to 4 turns of revision under Content$_1$ setting. We report the (top) checklist coverage (actual vs. oracle), (middle) incorporation rate, and (bottom) break rate.
• Figure 4: Coverage (Left), Break rate, and Incorporation rate (Right) with varying $k$. Break and incorporation rates are averaged across 5 DRAs since they all show the same trend.
• Figure 5: Negative-weight Reminder Text.