Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Retrieval-Augmented Generation for Natural Language Processing: A Survey

Shangyu Wu, Ying Xiong, Yufei Cui, Haolun Wu, Can Chen, Ye Yuan, Lianming Huang, Xue Liu, Tei-Wei Kuo, Nan Guan, Chun Jason Xue

TL;DR

<3-5 sentence high-level summary> Retrieval-Augmented Generation (RAG) addresses hallucinations and knowledge-update challenges in large language models by grounding generation in external, retrievable knowledge. The survey systematically dissects the retriever, retrieval fusions, and generators, and covers training strategies with and without datastore updates, evaluation benchmarks, and diverse NLP tasks and applications. It highlights practical tutorial codes, domain-specific benchmarks, and frameworks (e.g., LangChain, LLaMAIndex) that operationalize RAG in real-world systems. By comparing fusion modalities and outlining future directions, the paper offers a comprehensive roadmap for researchers and practitioners to design scalable, knowledge-grounded NLP systems.

Abstract

Large language models (LLMs) have demonstrated great success in various fields, benefiting from their huge amount of parameters that store knowledge. However, LLMs still suffer from several key issues, such as hallucination problems, knowledge update issues, and lacking domain-specific expertise. The appearance of retrieval-augmented generation (RAG), which leverages an external knowledge database to augment LLMs, makes up those drawbacks of LLMs. This paper reviews all significant techniques of RAG, especially in the retriever and the retrieval fusions. Besides, tutorial codes are provided for implementing the representative techniques in RAG. This paper further discusses the RAG update, including RAG with/without knowledge update. Then, we introduce RAG evaluation and benchmarking, as well as the application of RAG in representative NLP tasks and industrial scenarios. Finally, this paper discusses RAG's future directions and challenges for promoting this field's development.

Retrieval-Augmented Generation for Natural Language Processing: A Survey

TL;DR

<3-5 sentence high-level summary> Retrieval-Augmented Generation (RAG) addresses hallucinations and knowledge-update challenges in large language models by grounding generation in external, retrievable knowledge. The survey systematically dissects the retriever, retrieval fusions, and generators, and covers training strategies with and without datastore updates, evaluation benchmarks, and diverse NLP tasks and applications. It highlights practical tutorial codes, domain-specific benchmarks, and frameworks (e.g., LangChain, LLaMAIndex) that operationalize RAG in real-world systems. By comparing fusion modalities and outlining future directions, the paper offers a comprehensive roadmap for researchers and practitioners to design scalable, knowledge-grounded NLP systems.

Abstract

Large language models (LLMs) have demonstrated great success in various fields, benefiting from their huge amount of parameters that store knowledge. However, LLMs still suffer from several key issues, such as hallucination problems, knowledge update issues, and lacking domain-specific expertise. The appearance of retrieval-augmented generation (RAG), which leverages an external knowledge database to augment LLMs, makes up those drawbacks of LLMs. This paper reviews all significant techniques of RAG, especially in the retriever and the retrieval fusions. Besides, tutorial codes are provided for implementing the representative techniques in RAG. This paper further discusses the RAG update, including RAG with/without knowledge update. Then, we introduce RAG evaluation and benchmarking, as well as the application of RAG in representative NLP tasks and industrial scenarios. Finally, this paper discusses RAG's future directions and challenges for promoting this field's development.
Paper Structure (45 sections, 3 equations, 4 figures, 5 algorithms)

This paper contains 45 sections, 3 equations, 4 figures, 5 algorithms.

Table of Contents

  1. Introduction
  2. Overview of Retrieval-Augmented Generation
  3. Retriever
  4. Building the Retriever
  5. Chunking Corpus
  6. Encoding Chunks
  7. Building the Index
  8. Building the Datastore with Key-Value Pairs
  9. Code Demonstrations
  10. Querying the Retriever
  11. Encoding Queries and ANN Search
  12. Post-Processing
  13. Code Demonstrations
  14. Retrieval Fusions
  15. Query-based Fusion
  16. ...and 30 more sections

Figures (4)

  • Figure 1: The overview of retrieval-augmented generation for natural language processing. The inputs as queries are fed into both the retriever for retrieval knowledge and the generator for outputs. There are three kinds of retrieval fusions, including query-based fusion, logits-based fusion, and latent fusion.
  • Figure 2: Two stages of using the retriever.
  • Figure 3: The categories of fusion methods in RAG.
  • Figure 4: Different RAG training strategies with/without datastore update.