WildGraphBench: Benchmarking GraphRAG with Wild-Source Corpora

TL;DR

WildGraphBench introduces a wild-source GraphRAG benchmark by using Wikipedia reference pages as a long, noisy retrieval corpus and citation-grounded Wikipedia statements as gold facts. It constructs a three-tiered task suite—single-fact, multi-fact, and section-level summarization—with a statement-grounded evaluation framework to stress long-context multi-document reasoning. Experimental results show graph-based retrieval improves multi-fact aggregation but offers limited gains for simple lookups and struggles with broad summarization under wild noise, highlighting the need for more robust evidence acquisition and synthesis. The work provides a dataset, evaluation protocol, and insights into graph construction and retrieval budgeting to advance robust GraphRAG research in real-world, heterogeneous settings.

Abstract

Graph-based Retrieval-Augmented Generation (GraphRAG) organizes external knowledge as a hierarchical graph, enabling efficient retrieval and aggregation of scattered evidence across multiple documents. However, many existing benchmarks for GraphRAG rely on short, curated passages as external knowledge, failing to adequately evaluate systems in realistic settings involving long contexts and large-scale heterogeneous documents. To bridge this gap, we introduce WildGraphBench, a benchmark designed to assess GraphRAG performance in the wild. We leverage Wikipedia's unique structure, where cohesive narratives are grounded in long and heterogeneous external reference documents, to construct a benchmark reflecting real-word scenarios. Specifically, we sample articles across 12 top-level topics, using their external references as the retrieval corpus and citation-linked statements as ground truth, resulting in 1,100 questions spanning three levels of complexity: single-fact QA, multi-fact QA, and section-level summarization. Experiments across multiple baselines reveal that current GraphRAG pipelines help on multi-fact aggregation when evidence comes from a moderate number of sources, but this aggregation paradigm may overemphasize high-level statements at the expense of fine-grained details, leading to weaker performance on summarization tasks. Project page:https://github.com/BstWPY/WildGraphBench.

Figures (11)

• Figure 1: Why we use Wikipedia references as wild evidence. Wikipedia articles are concise summaries with citation-linked statements. The linked reference pages are often long, noisy, and heterogeneous (e.g., news sites, blogs, PDFs, and public reports). This mismatch makes evidence retrieval and verification harder.
• Figure 2: Example instances in WildGraphBench: (1) single-fact questions grounded by a single gold statement and one reference, (2) multi-fact questions requiring evidence aggregation across multiple statements/references, and (3) section-level summary questions evaluated at the statement level.
• Figure 3: Three-phase workflow of WildGraphBench after data collection. Phase 1:citation-aware statement extraction, producing the Wikipedia gold corpus. Phase 2: design single-fact, multi-fact, and section-level summary questions. Phase 3: evaluate with statement-grounded accuracy and statement-level precision/recall/F1.
• Figure 4: Graph Quality. We build graphs with LightRAG. Left: the fraction of isolated nodes (lower means better connectivity). Right: the average degree (higher means denser links).
• Figure 5: Impact of retrieval budget (top-$k$) on F1 score for summary questions. F1 increases as $k$ grows, then drops when $k$ is too large. It peaks at $k=8$.