Passage Segmentation of Documents for Extractive Question Answering

TL;DR

This work addresses the underexplored role of document chunking in Retrieval-Augmented Generation for open-domain QA, introducing the Logits-Guided Multi-Granular Chunker (LGMGC). The framework comprises two modules: Logits-Guided Chunker, which uses the end-of-sequence probability $p([EOS])$ to place semantic boundaries, and Multi-Granular Chunker, which recursively produces child chunks at sizes $\theta/2$ and $\theta/4$ for diverse granularity; the two modules are combined to yield top-$k$ parent chunks for synthesis. Across GutenQA and LongBench, LGMGC yields superior retrieval metrics and higher end-to-end F1 than existing baselines, with robustness across chunk size $\theta$ and lower computational cost than fully LLM-based chunking strategies. The approach enables more efficient, accurate RAG pipelines suitable for local deployment and practical document QA tasks, and points to future work in automatic QA evaluation metrics and broader deployment.

Abstract

Retrieval-Augmented Generation (RAG) has proven effective in open-domain question answering. However, the chunking process, which is essential to this pipeline, often receives insufficient attention relative to retrieval and synthesis components. This study emphasizes the critical role of chunking in improving the performance of both dense passage retrieval and the end-to-end RAG pipeline. We then introduce the Logits-Guided Multi-Granular Chunker (LGMGC), a novel framework that splits long documents into contextualized, self-contained chunks of varied granularity. Our experimental results, evaluated on two benchmark datasets, demonstrate that LGMGC not only improves the retrieval step but also outperforms existing chunking methods when integrated into a RAG pipeline.

Figures (1)

• Figure 1: Overview of the proposed method: Logits-Guided Multi-Granular Chunker. It utilizes the comprehension ability of LLMs to split documents into coherent parent chunks and split them into child chunks by Multi-Granular Module. The left figure shows how Multi-Granular Module works during the inference.