Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Wiki Live Challenge: Challenging Deep Research Agents with Expert-Level Wikipedia Articles

Shaohan Wang, Benfeng Xu, Licheng Zhang, Mingxuan Du, Chiwei Zhu, Xiaorui Wang, Zhendong Mao, Yongdong Zhang

TL;DR

This work introduces the Wiki Live Challenge (WLC), a live benchmark that uses recent Wikipedia Good Articles as expert references to evaluate Deep Research Agents (DRAs) on long-form, Wikipedia-style writing. The authors present Wiki Eval, a two-part framework comprising Wiki Writing (37–39 GA-derived criteria, evaluated by an LLM judge) and Wiki Fact (factual coverage relative to Wikipedia and source citations, verified by fact-checking pipelines). Across 100 GA spanning 15 domains, the study reveals a substantial gap between current DRAs and human expert-level Wikipedia articles, with notable variability across models, categories, and leakage scenarios. The benchmark and evaluation framework aim to provide reliable, fine-grained, and reproducible progress indicators for DRA research and development, with ongoing updates and data governance.

Abstract

Deep Research Agents (DRAs) have demonstrated remarkable capabilities in autonomous information retrieval and report generation, showing great potential to assist humans in complex research tasks. Current evaluation frameworks primarily rely on LLM-generated references or LLM-derived evaluation dimensions. While these approaches offer scalability, they often lack the reliability of expert-verified content and struggle to provide objective, fine-grained assessments of critical dimensions. To bridge this gap, we introduce Wiki Live Challenge (WLC), a live benchmark that leverages the newest Wikipedia Good Articles (GAs) as expert-level references. Wikipedia's strict standards for neutrality, comprehensiveness, and verifiability serve as a great challenge for DRAs, with GAs representing the pinnacle of which. We curate a dataset of 100 recent Good Articles and propose Wiki Eval, a comprehensive evaluation framework comprising a fine-grained evaluation method with 39 criteria for writing quality and rigorous metrics for factual verifiability. Extensive experiments on various DRA systems demonstrate a significant gap between current DRAs and human expert-level Wikipedia articles, validating the effectiveness of WLC in advancing agent research. We release our benchmark at https://github.com/WangShao2000/Wiki_Live_Challenge

Wiki Live Challenge: Challenging Deep Research Agents with Expert-Level Wikipedia Articles

TL;DR

This work introduces the Wiki Live Challenge (WLC), a live benchmark that uses recent Wikipedia Good Articles as expert references to evaluate Deep Research Agents (DRAs) on long-form, Wikipedia-style writing. The authors present Wiki Eval, a two-part framework comprising Wiki Writing (37–39 GA-derived criteria, evaluated by an LLM judge) and Wiki Fact (factual coverage relative to Wikipedia and source citations, verified by fact-checking pipelines). Across 100 GA spanning 15 domains, the study reveals a substantial gap between current DRAs and human expert-level Wikipedia articles, with notable variability across models, categories, and leakage scenarios. The benchmark and evaluation framework aim to provide reliable, fine-grained, and reproducible progress indicators for DRA research and development, with ongoing updates and data governance.

Abstract

Deep Research Agents (DRAs) have demonstrated remarkable capabilities in autonomous information retrieval and report generation, showing great potential to assist humans in complex research tasks. Current evaluation frameworks primarily rely on LLM-generated references or LLM-derived evaluation dimensions. While these approaches offer scalability, they often lack the reliability of expert-verified content and struggle to provide objective, fine-grained assessments of critical dimensions. To bridge this gap, we introduce Wiki Live Challenge (WLC), a live benchmark that leverages the newest Wikipedia Good Articles (GAs) as expert-level references. Wikipedia's strict standards for neutrality, comprehensiveness, and verifiability serve as a great challenge for DRAs, with GAs representing the pinnacle of which. We curate a dataset of 100 recent Good Articles and propose Wiki Eval, a comprehensive evaluation framework comprising a fine-grained evaluation method with 39 criteria for writing quality and rigorous metrics for factual verifiability. Extensive experiments on various DRA systems demonstrate a significant gap between current DRAs and human expert-level Wikipedia articles, validating the effectiveness of WLC in advancing agent research. We release our benchmark at https://github.com/WangShao2000/Wiki_Live_Challenge
Paper Structure (42 sections, 4 equations, 8 figures, 11 tables)

This paper contains 42 sections, 4 equations, 8 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Deep Research Agent
  4. Deep Research Benchmarks
  5. Wiki Live Challenge
  6. Data Construction
  7. Evaluation Methodology
  8. Wiki Writing
  9. Wiki Fact
  10. Experimental Settings
  11. Implementation Details
  12. Evaluated Models
  13. Results and Discussions
  14. Main Results
  15. Evaluation on Wiki Writing
  16. ...and 27 more sections

Figures (8)

  • Figure 1: Gap between human-written and AI-generated Wikipedia articles. Human-authored articles (top) feature rigorous citations and neutral tone, while AI-generated ones (bottom) lack citations and exhibit bias toward trending topics.
  • Figure 2: The overview of our Wiki Live Challenge (WLC) benchmark. (a) We continuously collect recent Wikipedia articles (e.g., from Mar. 1 to Dec. 1 in this iteration), filter the latest expert-reviewed Good Articles, and build the live task dataset. (b) We strictly grounded in GA criteria: well-written, neutral, broad coverage and verifiable. (c) Our evaluation framework, Wiki Eval, incorporates two key dimensions: Wiki Writing and Wiki Fact.
  • Figure 3: Overview of the WLC benchmark dataset. The left panel displays the distribution of collected Wikipedia Good Articles across 15 major categories and the key statistics of the WLC Benchmark Dataset. The right panel illustrates a representative task case.
  • Figure 4: Fact Coverage Heatmap on Wikipedia Good Article "Parasitic Ant". The x-axis represents individual facts ordered by their appearance in the article sections, and the y-axis represents different DRAs.
  • Figure 5: The six Wikipedia Good Article criteria. These criteria serve as the foundation for our Wiki Eval framework.
  • ...and 3 more figures