SagaScale: A Realistic, Scalable, and High-Quality Long-Context Benchmark Built from Full-Length Novels

TL;DR

SagaScale addresses the challenge of long-context understanding by building a realistic, scalable benchmark from full-length novels using an automated QA-generation pipeline grounded in external resources. It introduces a bilingual dataset with unprecedented context lengths and evaluates three long-context strategies (Long Context, Naïve RAG, Agentic RAG) across 12 frontier LLMs, offering detailed insights into performance, retrieval bottlenecks, and context-length effects. The paper contributes a high-quality data pipeline, a large, realistic benchmark, and an extensive analysis that informs long-context model design, with public release to accelerate progress. Overall, SagaScale demonstrates that directly supplying full context can outperform retrieval-based methods for capable models, while agentic retrieval helps mitigate retrieval bottlenecks, marking a step forward for evaluating and advancing long-context AI systems.

Abstract

Large Language Models (LLMs) have shown significant progress, but understanding long and complex documents remains challenging. Many long-context benchmarks have been proposed, but they face several limitations, including task realism, data scalability, and data quality. To this end, we introduce SagaScale, a realistic, scalable, and high-quality long-context benchmark built from full-length novels. The entire benchmark is constructed using an automated data collection pipeline that utilizes external resources (e.g., Wikipedia pages) to curate question-answer pairs. Critically, these external resources are provided only for benchmark construction and not during evaluation, which allows LLMs to curate complex questions that go beyond what they can answer during evaluation. SagaScale is also bilingual and offers the largest context length to date, with average token counts exceeding 250K for English novels and 320K for Chinese novels. Our evaluation across 12 frontier LLMs and three long-context methods -- Naïve RAG, Agentic RAG, and Long Context -- yields key insights, including: (1) Directly supplying the full context to the LLM can outperform other methods by a large margin; (2) Most LLMs still struggle with lengthy contexts, but Gemini-2.5-Pro stands out as an exception; and (3) Agentic RAG effectively addresses the retrieval bottleneck in Naïve RAG. Finally, we publicly release the SagaScale benchmark and our data collection codebase to facilitate future research.

Figures (18)

• Figure 1: Data collection pipeline of SagaScale.
• Figure 2: Distribution of questions across question types.
• Figure 3: Accuracy versus book length. Left (a): A comparison of accuracy decay for the three evaluated methods. Right (b): A breakdown of the Long Context performance for each model.
• Figure 4: Model accuracy as a function of tool-call chain length in Agentic RAG. All models show a decline in accuracy when more tool calls are invoked. For statistical validity, data points representing fewer than 10 samples are omitted.
• Figure 5: Violin plots illustrating the distribution of book token counts within the English (left) and Chinese (right) subsets of SagaScale.