Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarks

TL;DR

This work tackles the unreliability of IR benchmarks caused by unlabeled yet relevant chunks by introducing DREAM, a multi-agent debate framework that uses opposing stances and iterative critique to auto-label relevance with high accuracy and minimal human input ($95.2\%$) at $3.5\%$ human involvement. DREAM’s debate history serves as a shared resource for both automated labeling and human adjudication, enabling efficient escalation only for genuinely uncertain cases. Applying DREAM to BEIR and RobustQA, the BRIDGE benchmark uncovers $29{,}824$ holes, increasing gold chunks by $428\%$ and reducing evaluation bias, thereby enabling fairer retriever comparisons and more faithful RAG assessment. Re-benchmarking across 25 retrieval systems demonstrates that holes distort rankings and misalign retrieval with generation, while BRIDGE improves retrieval–generation alignment (RAGAlign@10) and shifts system rankings toward more accurate evaluations. Overall, the DREAM–BRIDGE pipeline provides scalable, high-quality relevance labeling and bias-reduced benchmarks that improve IR and RAG system analysis.

Abstract

Information retrieval (IR) evaluation remains challenging due to incomplete IR benchmark datasets that contain unlabeled relevant chunks. While LLMs and LLM-human hybrid strategies reduce costly human effort, they remain prone to LLM overconfidence and ineffective AI-to-human escalation. To address this, we propose DREAM, a multi-round debate-based relevance assessment framework with LLM agents, built on opposing initial stances and iterative reciprocal critique. Through our agreement-based debate, it yields more accurate labeling for certain cases and more reliable AI-to-human escalation for uncertain ones, achieving 95.2% labeling accuracy with only 3.5% human involvement. Using DREAM, we build BRIDGE, a refined benchmark that mitigates evaluation bias and enables fairer retriever comparison by uncovering 29,824 missing relevant chunks. We then re-benchmark IR systems and extend evaluation to RAG, showing that unaddressed holes not only distort retriever rankings but also drive retrieval-generation misalignment. The relevance assessment framework is available at https: //github.com/DISL-Lab/DREAM-ICLR-26; and the BRIDGE dataset is available at https://github.com/DISL-Lab/BRIDGE-Benchmark.

Figures (4)

• Figure 1: Overview of DREAM pipeline for constructing the BRIDGE benchmark. Multi-agent debate with opposing stances conducts a multi-round process, automatically annotating agreement cases and escalating disagreements to humans with debate history.
• Figure 2: Auto-labeling accuracy and escalation ratio of DREAM compared with two LLMJudge and LARA on non-escalated cases.
• Figure 3: Bias reduction via incremental addition of retrievals in DREAM’s hole filling: (a) reducing growth rate of newly identified holes across seven benchmark subsets as more systems are incorporated into the candidate pool; and (b) diminishing marginal contribution of a target retriever, measured by Hit@10, nDCG@10, and Recall@10, as the pool expands. To ensure statistical reliability, results are averaged over $10$ runs, each with 25 retrieval systems added in random order.
• Figure 4: Normalized retrieval (green) and generation (yellow) performance across RAG systems, using the same Llama3.1-8B-Instruct model but different retrievers (labeled by name). Systems are sorted in ascending order of Hit@10 scores along the x-axis. Colors indicate rank changes between the original benchmark and BRIDGE: Red for drops and Blue for improvements, from (a) to (b).