IDRBench: Interactive Deep Research Benchmark

TL;DR

IDRBench introduces the first interactive deep research benchmark, addressing the gap where real-world research goals are underspecified and evolve during exploration. By coupling a modular multi-agent framework (Planner, Supervisor, Researcher, Reporter) with an interaction mechanism (Evaluator, Questioner, User Simulator) and a scalable Ambiguity Injection data pipeline, IDRBench enables systematic evaluation of both the benefits and costs of interaction. The evaluation suite combines Report Similarity, Multi-Granularity F1-Score, and LLM-ACS to measure quality and alignment, alongside interaction cost metrics such as turns and tokens. Experiments across seven state-of-the-art LLMs show that interaction yields consistent quality gains and robustness improvements, with notable trade-offs in efficiency and cost that vary by model. Overall, IDRBench provides a principled framework for designing, evaluating, and deploying interactive deep research agents in real-world settings.

Abstract

Deep research agents powered by Large Language Models (LLMs) can perform multi-step reasoning, web exploration, and long-form report generation. However, most existing systems operate in an autonomous manner, assuming fully specified user intent and evaluating only final outputs. In practice, research goals are often underspecified and evolve during exploration, making sustained interaction essential for robust alignment. Despite its importance, interaction remains largely invisible to existing deep research benchmarks, which neither model dynamic user feedback nor quantify its costs. We introduce IDRBench, the first benchmark for systematically evaluating interactive deep research. IDRBench combines a modular multi-agent research framework with on-demand interaction, a scalable reference-grounded user simulator, and an interaction-aware evaluation suite that jointly measures interaction benefits (quality and alignment) and costs (turns and tokens). Experiments across seven state-of-the-art LLMs show that interaction consistently improves research quality and robustness, often outweighing differences in model capacity, while revealing substantial trade-offs in interaction efficiency.

Figures (6)

• Figure 1: Comparison of autonomous and interactive deep research. Autonomous agents execute independently and may diverge from user intent, while interactive agents incorporate feedback to maintain alignment.
• Figure 2: Overview of IDRBench. The benchmark integrates an interactive deep research framework with curated data construction, representative LLMs, and interaction-aware evaluation. It features a multi-agent pipeline for Planning, Research Loop, and Generation, augmented with an interaction mechanism for Clarification and User Feedback, and enables systematic evaluation of both interaction benefits and interaction costs.
• Figure 3: Distribution of average scores across seven LLMs, showing stability gains from interaction.
• Figure 4: Evaluator's prompt.
• Figure 5: Questioner's prompt.