Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Data

TL;DR

HoloBench addresses the challenge of holistic reasoning over massive textual data by reframing long-context evaluation as database-like inference over extended contexts. It systematically varies context length $|c|$, information density $d$, information distribution, and query complexity, using automated text-to-SQL generation to produce dynamic gold answers. Empirical results show that the amount of information in the context has a larger impact than mere context length, with query complexity—especially aggregation—being a major driver of difficulty; larger models (e.g., GPT-4o) tend to outperform in long contexts, while reasoning-focused LCLMs struggle as $|c|$ grows. CoT prompting significantly improves performance across contexts and query types, whereas RAG tends to offer limited or conditional benefits for holistic reasoning. Overall, HoloBench exposes both progress and enduring gaps in enabling robust holistic reasoning over long textual data and provides a reproducible framework for future improvements in long-context reasoning and prompting strategies.

Abstract

The rapid increase in textual information means we need more efficient methods to sift through, organize, and understand it all. While retrieval-augmented generation (RAG) models excel in accessing information from large document collections, they struggle with complex tasks that require aggregation and reasoning over information spanning across multiple documents--what we call holistic reasoning. Long-context language models (LCLMs) have great potential for managing large-scale documents, but their holistic reasoning capabilities remain unclear. In this work, we introduce HoloBench, a novel framework that brings database reasoning operations into text-based contexts, making it easier to systematically evaluate how LCLMs handle holistic reasoning across large documents. Our approach adjusts key factors such as context length, information density, distribution of information, and query complexity to evaluate LCLMs comprehensively. Our experiments show that the amount of information in the context has a bigger influence on LCLM performance than the actual context length. Furthermore, the complexity of queries affects performance more than the amount of information, particularly for different types of queries. Interestingly, queries that involve finding maximum or minimum values are easier for LCLMs and are less affected by context length, even though they pose challenges for RAG systems. However, tasks requiring the aggregation of multiple pieces of information show a noticeable drop in accuracy as context length increases. Additionally, we find that while grouping relevant information generally improves performance, the optimal positioning varies across models. Our findings surface both the advancements and the ongoing challenges in achieving a holistic understanding of long contexts.

Figures (16)

• Figure 1: Overview of the HoloBench instance generation process, illustrating the partitioning of databases into relevant and non-relevant subsets, verbalization of table rows into textual contexts, and construction of the final inference context based on information positioning parameters.
• Figure 2: Illustration of how relevant context $c_q$ (highlighted in green) is placed within the entire context $c$ for different information positioning strategies.
• Figure 3: Variations in model performances with position of relevant information. Concentrated information benefits most models, although optimal position varies for each model.
• Figure 4: LCLMs' performance on various query types. Max/Min queries are an easier task across context lengths than other types. In contrast, We observe a performance drop for Aggregation queries, particularly for longer contexts.
• Figure 5: LCLMs' performance on various query difficulties. Model performance declines as query difficulty increases. Performance drops further as context length increases.