Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Systematic Analysis of Chunking Strategies for Reliable Question Answering

Sofia Bennani, Charles Moslonka

TL;DR

The paper addresses how chunking decisions affect the reliability and cost of industrial Retrieval-Augmented Generation (RAG) systems. It conducts a systematic, end-to-end evaluation across chunking methods, sizes, overlaps, and context budgets using a SPLADE-based sparse retriever and a Mistral-8B-Instruct generator on Natural Questions. Key contributions include practical defaults—no overlap, sentence chunking, and task-tuned context budgets—along with a characterization of the context cliff around $C \approx 2.5k$ tokens and abstention behavior to balance semantic fidelity and factual accuracy. The findings offer actionable guidance for cost-efficient, robust RAG deployment in production environments and establish baselines for future IR-for-agents evaluations.

Abstract

We study how document chunking choices impact the reliability of Retrieval-Augmented Generation (RAG) systems in industry. While practice often relies on heuristics, our end-to-end evaluation on Natural Questions systematically varies chunking method (token, sentence, semantic, code), chunk size, overlap, and context length. We use a standard industrial setup: SPLADE retrieval and a Mistral-8B generator. We derive actionable lessons for cost-efficient deployment: (i) overlap provides no measurable benefit and increases indexing cost; (ii) sentence chunking is the most cost-effective method, matching semantic chunking up to ~5k tokens; (iii) a "context cliff" reduces quality beyond ~2.5k tokens; and (iv) optimal context depends on the goal (semantic quality peaks at small contexts; exact match at larger ones).

A Systematic Analysis of Chunking Strategies for Reliable Question Answering

TL;DR

The paper addresses how chunking decisions affect the reliability and cost of industrial Retrieval-Augmented Generation (RAG) systems. It conducts a systematic, end-to-end evaluation across chunking methods, sizes, overlaps, and context budgets using a SPLADE-based sparse retriever and a Mistral-8B-Instruct generator on Natural Questions. Key contributions include practical defaults—no overlap, sentence chunking, and task-tuned context budgets—along with a characterization of the context cliff around tokens and abstention behavior to balance semantic fidelity and factual accuracy. The findings offer actionable guidance for cost-efficient, robust RAG deployment in production environments and establish baselines for future IR-for-agents evaluations.

Abstract

We study how document chunking choices impact the reliability of Retrieval-Augmented Generation (RAG) systems in industry. While practice often relies on heuristics, our end-to-end evaluation on Natural Questions systematically varies chunking method (token, sentence, semantic, code), chunk size, overlap, and context length. We use a standard industrial setup: SPLADE retrieval and a Mistral-8B generator. We derive actionable lessons for cost-efficient deployment: (i) overlap provides no measurable benefit and increases indexing cost; (ii) sentence chunking is the most cost-effective method, matching semantic chunking up to ~5k tokens; (iii) a "context cliff" reduces quality beyond ~2.5k tokens; and (iv) optimal context depends on the goal (semantic quality peaks at small contexts; exact match at larger ones).
Paper Structure (15 sections, 2 figures, 1 table)

This paper contains 15 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Experimental Setup
  3. Task and corpus.
  4. Retrieval and context budgeting.
  5. Chunking strategies.
  6. Generation and abstention.
  7. Parameters varied.
  8. Metrics and protocol.
  9. Findings
  10. F1. Overlap adds cost without measurable gains.
  11. F2. Method “tier list”: sentence $\approx$ semantic > token $\gg$ code (for text).
  12. F3. The “context cliff”: more is not always better.
  13. F4. Goal-driven tuning: small $C$ for semantic quality; larger $C$ for factual accuracy; abstention is tunable.
  14. Limitations and Future Work.
  15. Conclusion

Figures (2)

  • Figure 1: RAG pipeline architecture with parameters $S, O$ and $C$.
  • Figure 2: Left: Effect of context length $C$ on metrics for different chunking methods (Sentence, Semantic, Token). Right: Chunking method comparison at fixed $C=5000$ tokens and $S=300$, $O=0$. Dots show means; bars denote 95% bootstrap CIs.