A New HOPE: Domain-agnostic Automatic Evaluation of Text Chunking

TL;DR

This work investigates how the way documents are chunked into passages affects Retrieval-Augmented Generation (RAG) performance, highlighting that LLMs are sensitive to data layout and structure. It introduces HOPE, a domain-agnostic automatic evaluation that quantifies chunking quality across intrinsic passage properties, extrinsic passage properties, and passages-document coherence, without requiring annotated data. Across seven domains, HOPE correlates with RAG performance, with semantic independence driving substantial gains in factual and answer correctness, while concept unity shows minimal or even negative impact. The findings offer actionable guidance for designing chunking strategies and suggest future directions such as reward-based chunk boundary optimization and domain-adaptive HOPE applications to improve factuality and response quality in RAG systems.

Abstract

Document chunking fundamentally impacts Retrieval-Augmented Generation (RAG) by determining how source materials are segmented before indexing. Despite evidence that Large Language Models (LLMs) are sensitive to the layout and structure of retrieved data, there is currently no framework to analyze the impact of different chunking methods. In this paper, we introduce a novel methodology that defines essential characteristics of the chunking process at three levels: intrinsic passage properties, extrinsic passage properties, and passages-document coherence. We propose HOPE (Holistic Passage Evaluation), a domain-agnostic, automatic evaluation metric that quantifies and aggregates these characteristics. Our empirical evaluations across seven domains demonstrate that the HOPE metric correlates significantly (p > 0.13) with various RAG performance indicators, revealing contrasts between the importance of extrinsic and intrinsic properties of passages. Semantic independence between passages proves essential for system performance with a performance gain of up to 56.2% in factual correctness and 21.1% in answer correctness. On the contrary, traditional assumptions about maintaining concept unity within passages show minimal impact. These findings provide actionable insights for optimizing chunking strategies, thus improving RAG system design to produce more factually correct responses.

Figures (7)

• Figure 1: The three levels of holistic chunking evaluation. Left: The intrinsic passage properties are concerned with the information isolated within a passage. Middle: Extrinsic passage properties relate to how passages impact each other. Right: Passages-document coherence focuses on how well the passage set represents the original document.
• Figure 2: Calculating the concept unity $\Bar{\zeta}_{con}$: The passage in green is forwarded to an LLM to produce a set of statements $\mathcal{S}$. The statements are then transformed into vector representations using an embedding model. The pairwise cosine similarity between the vectors is calculated.
• Figure 3: Calculation of the semantic independence $\Bar{\zeta}_{sem}$: A selected passage $p^*$ is used to construct a set of questions $Q$. The questions are then answered by two LLMs: one that can access all passages $\mathcal{P}$ (RAG) and one that can only access the focus passage $p^*$ (Completion). The two LLMs produce the answers $\mathcal{A}$ and $\mathcal{A}^*$, which are then compared using an embedding model and pairwise cosine similarity.
• Figure 4: Calculation of the information preservation $\zeta_{inf}$: Three-sentence segments are randomly sampled from the original document. An LLM uses the segments to construct statement quadruplets, where one is verifiable true and three are plausible but false. A secondary LLM is then tasked with truth discrimination by analyzing the statements against contextually relevant passages.
• Figure 5: Documents and questions from diverse domains are processed through eight chunking configurations (four fixed-size, two recursive, and two semantic chunking variants). For each resulting passage collection, both the HOPE metric and RAGAS scores are computed. The setup then analyzes correlation between HOPE and RAGAS scores across all configurations and datasets.

Theorems & Definitions (1)

• Definition 1: Chunking