CorpusQA: A 10 Million Token Benchmark for Corpus-Level Analysis and Reasoning

TL;DR

CorpusQA targets corpus-level reasoning by evaluating how models reason over large, dispersed document repositories. It introduces a six-stage data-generation pipeline that produces guaranteed-ground-truth long-context QA pairs by first structuring data, generating diverse queries, and deriving answers via NL2SQL on a global data table, then re-presenting unstructured documents as the final context. Experiments show standard LLMs struggle as context grows, RAG systems fail under heavy dispersion, and memory-augmented agents offer the most robust path forward. The study also demonstrates that training on synthesized data improves long-context reasoning and can generalize to other corpus-scale benchmarks, underscoring the need for architectures that perform global information synthesis rather than merely expanding context windows.

Abstract

While large language models now handle million-token contexts, their capacity for reasoning across entire document repositories remains largely untested. Existing benchmarks are inadequate, as they are mostly limited to single long texts or rely on a "sparse retrieval" assumption-that answers can be derived from a few relevant chunks. This assumption fails for true corpus-level analysis, where evidence is highly dispersed across hundreds of documents and answers require global integration, comparison, and statistical aggregation. To address this critical gap, we introduce CorpusQA, a new benchmark scaling up to 10 million tokens, generated via a novel data synthesis framework. By decoupling reasoning from textual representation, this framework creates complex, computation-intensive queries with programmatically guaranteed ground-truth answers, challenging systems to perform holistic reasoning over vast, unstructured text without relying on fallible human annotation. We further demonstrate the utility of our framework beyond evaluation, showing that fine-tuning on our synthesized data effectively enhances an LLM's general long-context reasoning capabilities. Extensive experiments reveal that even state-of-the-art long-context LLMs struggle as input length increases, and standard retrieval-augmented generation systems collapse entirely. Our findings indicate that memory-augmented agentic architectures offer a more robust alternative, suggesting a critical shift is needed from simply extending context windows to developing advanced architectures for global information synthesis.

Figures (4)

• Figure 1: RAG-style benchmarks assume sparse retrieval, where the answer can be derived from a few retrieved documents. In contrast, corpus-level benchmarks require holistic reasoning over an entire document repository with highly dispersed evidence.
• Figure 2: Overall data generation pipeline of CorpusQA. The process consists of six steps: (1) document collection and filtering to ensure long and data-rich inputs, (2) schema extraction for structured representation, (3) query generation via templates and LLM-based expansion, (4) data table aggregation across documents, (5) programmatic ground-truth generation using NL2SQL execution, and (6) final QA pair assembly by concatenating original documents with generated queries and answers. This design guarantees both realism and verifiable correctness for corpus-level reasoning tasks.
• Figure 3: Impact of fine-tuning with synthesized data. Training a model on data generated by our synthesis framework not only improves in-domain performance (CorpusQA) but also generalizes to enhance performance on external long-context benchmarks (LongBenchV2 and FRAMES).
• Figure 4: Comparison of Thinking vs. Instruct models on the 128K setting. Reasoning-oriented models (Thinking) consistently outperform their instruction-tuned counterparts, highlighting the benchmark's sensitivity to complex reasoning over simple instruction following.