LiveResearchBench: A Live Benchmark for User-Centric Deep Research in the Wild

TL;DR

This work introduces LiveResearchBench, a dynamic, user-centric benchmark of 100 expert-curated tasks spanning diverse domains to assess deep-research agents that search real-time web sources. It pairs LiveResearchBench with DeepEval, a comprehensive six-dimension evaluation suite and multi-protocol evaluation framework to robustly judge long-form, citation-grounded reports. Through a large-scale evaluation of 17 frontier systems (single-agent and multi-agent), the authors reveal that while current systems excel at information gathering and presentation, they struggle with citation reliability, factual-consistent synthesis, and depth of analysis. The results underscore the need for advanced memory, information compression, and synthesis modules to realize reliable, insightful deep research. The work provides a rigorous, reusable benchmark-and-evaluation backbone to drive improvements in agentic research systems.

Abstract

Deep research -- producing comprehensive, citation-grounded reports by searching and synthesizing information from hundreds of live web sources -- marks an important frontier for agentic systems. To rigorously evaluate this ability, four principles are essential: tasks should be (1) user-centric, reflecting realistic information needs, (2) dynamic, requiring up-to-date information beyond parametric knowledge, (3) unambiguous, ensuring consistent interpretation across users, and (4) multi-faceted and search-intensive, requiring search over numerous web sources and in-depth analysis. Existing benchmarks fall short of these principles, often focusing on narrow domains or posing ambiguous questions that hinder fair comparison. Guided by these principles, we introduce LiveResearchBench, a benchmark of 100 expert-curated tasks spanning daily life, enterprise, and academia, each requiring extensive, dynamic, real-time web search and synthesis. Built with over 1,500 hours of human labor, LiveResearchBench provides a rigorous basis for systematic evaluation. To evaluate citation-grounded long-form reports, we introduce DeepEval, a comprehensive suite covering both content- and report-level quality, including coverage, presentation, citation accuracy and association, consistency and depth of analysis. DeepEval integrates four complementary evaluation protocols, each designed to ensure stable assessment and high agreement with human judgments. Using LiveResearchBench and DeepEval, we conduct a comprehensive evaluation of 17 frontier deep research systems, including single-agent web search, single-agent deep research, and multi-agent systems. Our analysis reveals current strengths, recurring failure modes, and key system components needed to advance reliable, insightful deep research. Our code is available at: https://github.com/SalesforceAIResearch/LiveResearchBench.

Figures (22)

• Figure 1: Comparison of LiveResearchBench with existing deep research benchmarks. {{date}} will be replaced by the evaluation date. LiveResearchBench is explicitly user-centric, unambiguous, and time-varying, featuring multi-faceted, search-intensive tasks across diverse domains.
• Figure 2: Domain distribution and task coverage of LiveResearchBench.
• Figure 3: Six-stage data generation pipeline for LiveResearchBench. We begin with user interviews and surveys to capture realistic research needs, followed by expert-drafted research questions aligned with the identified domains. Clarification questions from frontier LLMs help ensure unambiguous scope, which human experts then refine into finalized queries. Finally, GPT-5 generates checklists that decompose each query into verifiable unit tests, enabling consistent coverage evaluation across systems. The quality of these checklists is further validated by human experts (Figure \ref{['fig:verify_pipeline']}).
• Figure 4: Five-stage data verification pipeline. Research questions and checklists are first independently assessed by expert annotators under detailed guidelines. Two rounds of quality control then ensure agreement across annotators and detect systematic issues. Finally, a separate group of experts conducts verification and cross-checking to resolve conflicts and finalize the dataset.
• Figure 5: Illustration of rubric tree–based evaluation of citation accuracy. Compared to evaluating all claim–URL pairs individually, the rubric tree groups claims by link, which significantly reduces evaluation costs while enabling finer-grained categorization of citation errors: invalid links (E1), irrelevant links (E2), and unsupported claims (E3).