Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ChunkRAG: Novel LLM-Chunk Filtering Method for RAG Systems

Ishneet Sukhvinder Singh, Ritvik Aggarwal, Ibrahim Allahverdiyev, Muhammad Taha, Aslihan Akalin, Kevin Zhu, Sean O'Brien

TL;DR

ChunkRAG introduces an LLM-driven chunk-filtering framework for RAG systems that operates at the granularity of text chunks rather than whole documents. By semantically chunking documents, embedding chunks, and applying a multi-stage relevance scoring pipeline—including self-reflection and critic models—the method filters out irrelevant content before generation, reducing hallucinations and improving factual accuracy. Empirical results on PopQA, PubHealth, and Biography demonstrate superior performance over standard and advanced RAG baselines, with notable gains in short, fact-heavy tasks. The approach emphasizes robust retrieval, dynamic thresholding, and constrained generation, offering a practical enhancement for knowledge-intensive applications like fact-checking and multi-hop reasoning.

Abstract

Retrieval-Augmented Generation (RAG) systems using large language models (LLMs) often generate inaccurate responses due to the retrieval of irrelevant or loosely related information. Existing methods, which operate at the document level, fail to effectively filter out such content. We propose LLM-driven chunk filtering, ChunkRAG, a framework that enhances RAG systems by evaluating and filtering retrieved information at the chunk level. Our approach employs semantic chunking to divide documents into coherent sections and utilizes LLM-based relevance scoring to assess each chunk's alignment with the user's query. By filtering out less pertinent chunks before the generation phase, we significantly reduce hallucinations and improve factual accuracy. Experiments show that our method outperforms existing RAG models, achieving higher accuracy on tasks requiring precise information retrieval. This advancement enhances the reliability of RAG systems, making them particularly beneficial for applications like fact-checking and multi-hop reasoning.

ChunkRAG: Novel LLM-Chunk Filtering Method for RAG Systems

TL;DR

ChunkRAG introduces an LLM-driven chunk-filtering framework for RAG systems that operates at the granularity of text chunks rather than whole documents. By semantically chunking documents, embedding chunks, and applying a multi-stage relevance scoring pipeline—including self-reflection and critic models—the method filters out irrelevant content before generation, reducing hallucinations and improving factual accuracy. Empirical results on PopQA, PubHealth, and Biography demonstrate superior performance over standard and advanced RAG baselines, with notable gains in short, fact-heavy tasks. The approach emphasizes robust retrieval, dynamic thresholding, and constrained generation, offering a practical enhancement for knowledge-intensive applications like fact-checking and multi-hop reasoning.

Abstract

Retrieval-Augmented Generation (RAG) systems using large language models (LLMs) often generate inaccurate responses due to the retrieval of irrelevant or loosely related information. Existing methods, which operate at the document level, fail to effectively filter out such content. We propose LLM-driven chunk filtering, ChunkRAG, a framework that enhances RAG systems by evaluating and filtering retrieved information at the chunk level. Our approach employs semantic chunking to divide documents into coherent sections and utilizes LLM-based relevance scoring to assess each chunk's alignment with the user's query. By filtering out less pertinent chunks before the generation phase, we significantly reduce hallucinations and improve factual accuracy. Experiments show that our method outperforms existing RAG models, achieving higher accuracy on tasks requiring precise information retrieval. This advancement enhances the reliability of RAG systems, making them particularly beneficial for applications like fact-checking and multi-hop reasoning.

Paper Structure

This paper contains 26 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Addressing Hallucinations in Large Language Models
  4. Retrieval-Augmented Generation Techniques
  5. Query Rewriting for Enhanced Retrieval
  6. Redundancy Reduction with Cosine Similarity
  7. Methodology
  8. Experiments
  9. Tasks, Datasets and Metrics
  10. Baselines
  11. Baselines Without Retrieval
  12. Baselines With Retrieval
  13. Analysis
  14. Comparison
  15. Insights
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Comparison of Response Generation With and Without Chunk Filtering
  • Figure 2: ChunkRAG Methodology for Enhanced Retrieval and Filtering. This figure illustrates the ChunkRAG pipeline, combining semantic chunking, filtering, and ensemble retrieval to optimize information relevance and accuracy, with final results re-ranked for precision.
  • Figure 3: Chunk Reduction vs. Similarity Threshold
  • Figure 4: Average Chunk Similarity Before/After Filtering