Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Classifying and Addressing the Diversity of Errors in Retrieval-Augmented Generation Systems

Kin Kwan Leung, Mouloud Belbahri, Yi Sui, Alex Labach, Xueying Zhang, Stephen Rose, Jesse C. Cresswell

TL;DR

This work addresses the diverse, real-world error modes in retrieval-augmented generation by introducing a comprehensive, stage-aligned taxonomy of RAG errors, paired with an auto-evaluation framework and a manually annotated error dataset. It groundss the taxonomy in a realistic modular RAG pipeline (chunking, retrieval, reranking, generation) and demonstrates cross-stage error characterization, including mitigation strategies for each class. Central to the contribution is RAGEC, a three-step system that evaluates answers, classifies the responsible pipeline stage, and identifies specific error types, along with a dataset of annotations to guide debugging and improvements. The study’s findings suggest that meaningful robustness gains can be achieved by targeted changes at chunking, retrieval, and reranking stages, complemented by generation-time safeguards, and it provides a practical resource for practitioners to triage and fix RAG pipelines. Overall, the framework supports more reliable, informed deployment of RAG systems in complex, real-world domains.

Abstract

Retrieval-augmented generation (RAG) is a prevalent approach for building LLM-based question-answering systems that can take advantage of external knowledge databases. Due to the complexity of real-world RAG systems, there are many potential causes for erroneous outputs. Understanding the range of errors that can occur in practice is crucial for robust deployment. We present a new taxonomy of the error types that can occur in realistic RAG systems, examples of each, and practical advice for addressing them. Additionally, we curate a dataset of erroneous RAG responses annotated by error types. We then propose an auto-evaluation method aligned with our taxonomy that can be used in practice to track and address errors during development. Code and data are available at https://github.com/layer6ai-labs/rag-error-classification.

Classifying and Addressing the Diversity of Errors in Retrieval-Augmented Generation Systems

TL;DR

This work addresses the diverse, real-world error modes in retrieval-augmented generation by introducing a comprehensive, stage-aligned taxonomy of RAG errors, paired with an auto-evaluation framework and a manually annotated error dataset. It groundss the taxonomy in a realistic modular RAG pipeline (chunking, retrieval, reranking, generation) and demonstrates cross-stage error characterization, including mitigation strategies for each class. Central to the contribution is RAGEC, a three-step system that evaluates answers, classifies the responsible pipeline stage, and identifies specific error types, along with a dataset of annotations to guide debugging and improvements. The study’s findings suggest that meaningful robustness gains can be achieved by targeted changes at chunking, retrieval, and reranking stages, complemented by generation-time safeguards, and it provides a practical resource for practitioners to triage and fix RAG pipelines. Overall, the framework supports more reliable, informed deployment of RAG systems in complex, real-world domains.

Abstract

Retrieval-augmented generation (RAG) is a prevalent approach for building LLM-based question-answering systems that can take advantage of external knowledge databases. Due to the complexity of real-world RAG systems, there are many potential causes for erroneous outputs. Understanding the range of errors that can occur in practice is crucial for robust deployment. We present a new taxonomy of the error types that can occur in realistic RAG systems, examples of each, and practical advice for addressing them. Additionally, we curate a dataset of erroneous RAG responses annotated by error types. We then propose an auto-evaluation method aligned with our taxonomy that can be used in practice to track and address errors during development. Code and data are available at https://github.com/layer6ai-labs/rag-error-classification.
Paper Structure (35 sections, 4 figures, 6 tables)

This paper contains 35 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. RAG Implementation
  4. Error Classification
  5. Chunking
  6. Retrieval
  7. Reranking
  8. Generation
  9. RAGEC: RAG Error Classification
  10. Conclusion
  11. RAG Pipeline and Dataset Details
  12. Chunker.
  13. Retrieval.
  14. Reranker.
  15. Generator.
  16. ...and 20 more sections

Figures (4)

  • Figure 1: Diagram of our implemented RAG architecture, reflective of systems currently applied in industry. Components are annotated with error types that are caused by them.
  • Figure 2: Mode error type distribution per dataset.
  • Figure 3: Joint distribution of first and second mode error categories for the generation stage. Left: Dragonball-EN Right: CLAPnq
  • Figure 4: Joint distribution of RAGEC auto-evaluation error type classifications and human annotations for DragonBall-EN