Deep Research Comparator: A Platform For Fine-grained Human Annotations of Deep Research Agents

TL;DR

Deep Research Comparator tackles the challenge of evaluating deep research agents that autonomously search, analyze, and generate long-form reports. It introduces a holistic, open-source platform that hosts agents, facilitates side-by-side comparisons, and collects fine-grained feedback on intermediate steps and text spans. A Simple Deepresearch baseline scaffold enables easy integration of various LLMs for evaluation. In a proof-of-concept with 176 queries and 17 annotators, the platform shows alignment with static benchmarks for overall report quality while revealing valuable distinctions in fine-grained signals, supporting richer diagnostics and potential process-level training such as RLHF.

Abstract

Effectively evaluating deep research agents that autonomously search the web, analyze information, and generate reports remains a major challenge, particularly when it comes to assessing long reports and giving detailed feedback on their intermediate steps. To address these gaps, we introduce Deep Research Comparator, a platform that offers a holistic framework for deep research agent hosting, side-by-side comparison, fine-grained human feedback collection, and ranking calculation. Given a user query, our platform displays the final reports from two different agents along with their intermediate steps during generation. Annotators can evaluate the overall quality of final reports based on side-by-side comparison, and also provide detailed feedback separately by assessing intermediate steps or specific text spans within the final report. Furthermore, we develop Simple Deepresearch, an end-to-end agent scaffold. This scaffold serves as a baseline that facilitates the easy integration of various large language models to transform them into deep research agents for evaluation. To demonstrate the platform's utility for deep research agent development, we have collected real user preference data from 17 annotators on three deep research agents. A demo video of our platform can be found at https://www.youtube.com/watch?v=g4d2dnbdseg.

Figures (4)

• Figure 1: Left: the main interface of Deep Research Comparator. Right: the fine-grained annotation mechanism for upvoting/ downvoting intermediate steps and text spans in report.
• Figure 2: High-level overview of Deep Research Comparator's system architecture.
• Figure 3: Win Probability of Agent A over Agent B.
• Figure 4: Overview of Simple Deepresearch, our Baseline agent scaffold.