Is Semantic Chunking Worth the Computational Cost?

TL;DR

This study systematically evaluates the effectiveness of semantic chunking using three common retrieval-related tasks: document retrieval, evidence retrieval, and retrieval-based answer generation, showing that the computational costs associated with semantic chunking are not justified by consistent performance gains.

Abstract

Recent advances in Retrieval-Augmented Generation (RAG) systems have popularized semantic chunking, which aims to improve retrieval performance by dividing documents into semantically coherent segments. Despite its growing adoption, the actual benefits over simpler fixed-size chunking, where documents are split into consecutive, fixed-size segments, remain unclear. This study systematically evaluates the effectiveness of semantic chunking using three common retrieval-related tasks: document retrieval, evidence retrieval, and retrieval-based answer generation. The results show that the computational costs associated with semantic chunking are not justified by consistent performance gains. These findings challenge the previous assumptions about semantic chunking and highlight the need for more efficient chunking strategies in RAG systems.

Figures (9)

• Figure 1: Illustration of the three chunkers tested in this study. Colored segments represent different topics within the sample document: Purple for psychology, Green for programming, and Yellow for food. Red blocks mark chunk breakpoints. (a) Fixed-size Chunker splits the document into consecutive, uniform chunks without considering semantic content. (b) Breakpoint-based Semantic Chunker segments the text by detecting semantic distance thresholds between consecutive sentences to maintain coherence. (c) Clustering-based Semantic Chunker groups semantically similar sentences, potentially combining non-consecutive text to form topic-based chunks.
• Figure 2: Performance vs. hyperparameter values for Clustering-based Semantic Chunker (Single-linkage). Left: Document Retrieval; Right: Evidence Retrieval. The x-axis shows n$\_$clusters, and the y-axis shows the metric value. Each subplot’s y-label indicates the fixed hyperparameter value, with $\lambda$ increasing from top to bottom.
• Figure 3: Performance vs. hyperparameter values for Clustering-based Semantic Chunker (DBSCAN). Left: Document Retrieval; Right: Evidence Retrieval. The x-axis shows eps, and the y-axis shows the metric value. Each subplot’s y-label indicates the fixed hyperparameter value, with $\lambda$ increasing from top to bottom.
• Figure 4: Performance vs. hyperparameter values for Breakpoint-based Semantic Chunker. Left: Document Retrieval; Right: Evidence Retrieval. The x-axis shows n$\_$clusters, and the y-axis shows the metric value. Each subplot’s y-label indicates the breakpoint threshold type.
• Figure 5: Performance vs. hyperparameter values for Fixed-size Chunker. Left: Document Retrieval; Right: Evidence Retrieval. The x-axis shows n$\_$chunks, and the y-axis shows the metric value. Each subplot’s y-label indicates the fixed hyperparameter value, with n_sents_overlap increasing from top to bottom.