Reconstructing Context: Evaluating Advanced Chunking Strategies for Retrieval-Augmented Generation

TL;DR

The paper investigates how advanced chunking strategies affect retrieval-augmented generation under realistic input constraints. It compares late chunking, which preserves global context before segmentation, with contextual retrieval, which augments each chunk with document-wide context via prompting and rank fusion, within a unified evaluation using diverse embedding models and datasets.Key findings show that contextual retrieval generally preserves semantic coherence and benefits from reranking, albeit at higher computational cost, while late chunking offers efficiency with variable performance across datasets and models. The work provides practical guidance for deploying RAG systems under resource constraints and contributes open-source materials to support reproducibility.

Abstract

Retrieval-augmented generation (RAG) has become a transformative approach for enhancing large language models (LLMs) by grounding their outputs in external knowledge sources. Yet, a critical question persists: how can vast volumes of external knowledge be managed effectively within the input constraints of LLMs? Traditional methods address this by chunking external documents into smaller, fixed-size segments. While this approach alleviates input limitations, it often fragments context, resulting in incomplete retrieval and diminished coherence in generation. To overcome these shortcomings, two advanced techniques, late chunking and contextual retrieval, have been introduced, both aiming to preserve global context. Despite their potential, their comparative strengths and limitations remain unclear. This study presents a rigorous analysis of late chunking and contextual retrieval, evaluating their effectiveness and efficiency in optimizing RAG systems. Our results indicate that contextual retrieval preserves semantic coherence more effectively but requires greater computational resources. In contrast, late chunking offers higher efficiency but tends to sacrifice relevance and completeness.

Figures (2)

• Figure 1: Comparison of early chunking (left) and late chunking (right) approaches for processing long documents. In early chunking, the document is divided into chunks before embedding, with each chunk processed independently by the embedding model and then pooled. In contrast, late chunking processes the entire document to generate token embeddings first, using boundary cues to create chunk embeddings, which are subsequently pooled.
• Figure 2: Contextualization of each chunk is performed prior to embedding. The document is divided into chunks, and a prompt is used to query an LLM to generate contextual information from the document for each chunk. The context is prepended to the chunk, which is then processed by the embedding model to produce the final chunk embedding.