Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing

Yucheng Hu, Yuxing Lu

TL;DR

RAG and RAU survey Retrieval-Augmented Language Models (RALMs), detailing interaction modes, retriever and LM taxonomies, data sources, enhancement strategies, and evaluation. It categorizes retrievers into Sparse, Dense, Internet, and Hybrid, and LMs into AutoEncoder, AutoRegressive, and Encoder-Decoder, discussing how these components interact and be improved. The paper highlights limitations in robustness, retrieval quality, and cost, and outlines future directions for broader applications and more efficient, reliable systems. A GitHub resource accompanies the survey to provide access to surveyed works and resources for researchers.

Abstract

Large Language Models (LLMs) have catalyzed significant advancements in Natural Language Processing (NLP), yet they encounter challenges such as hallucination and the need for domain-specific knowledge. To mitigate these, recent methodologies have integrated information retrieved from external resources with LLMs, substantially enhancing their performance across NLP tasks. This survey paper addresses the absence of a comprehensive overview on Retrieval-Augmented Language Models (RALMs), both Retrieval-Augmented Generation (RAG) and Retrieval-Augmented Understanding (RAU), providing an in-depth examination of their paradigm, evolution, taxonomy, and applications. The paper discusses the essential components of RALMs, including Retrievers, Language Models, and Augmentations, and how their interactions lead to diverse model structures and applications. RALMs demonstrate utility in a spectrum of tasks, from translation and dialogue systems to knowledge-intensive applications. The survey includes several evaluation methods of RALMs, emphasizing the importance of robustness, accuracy, and relevance in their assessment. It also acknowledges the limitations of RALMs, particularly in retrieval quality and computational efficiency, offering directions for future research. In conclusion, this survey aims to offer a structured insight into RALMs, their potential, and the avenues for their future development in NLP. The paper is supplemented with a Github Repository containing the surveyed works and resources for further study: https://github.com/2471023025/RALM_Survey.

RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing

TL;DR

RAG and RAU survey Retrieval-Augmented Language Models (RALMs), detailing interaction modes, retriever and LM taxonomies, data sources, enhancement strategies, and evaluation. It categorizes retrievers into Sparse, Dense, Internet, and Hybrid, and LMs into AutoEncoder, AutoRegressive, and Encoder-Decoder, discussing how these components interact and be improved. The paper highlights limitations in robustness, retrieval quality, and cost, and outlines future directions for broader applications and more efficient, reliable systems. A GitHub resource accompanies the survey to provide access to surveyed works and resources for researchers.

Abstract

Large Language Models (LLMs) have catalyzed significant advancements in Natural Language Processing (NLP), yet they encounter challenges such as hallucination and the need for domain-specific knowledge. To mitigate these, recent methodologies have integrated information retrieved from external resources with LLMs, substantially enhancing their performance across NLP tasks. This survey paper addresses the absence of a comprehensive overview on Retrieval-Augmented Language Models (RALMs), both Retrieval-Augmented Generation (RAG) and Retrieval-Augmented Understanding (RAU), providing an in-depth examination of their paradigm, evolution, taxonomy, and applications. The paper discusses the essential components of RALMs, including Retrievers, Language Models, and Augmentations, and how their interactions lead to diverse model structures and applications. RALMs demonstrate utility in a spectrum of tasks, from translation and dialogue systems to knowledge-intensive applications. The survey includes several evaluation methods of RALMs, emphasizing the importance of robustness, accuracy, and relevance in their assessment. It also acknowledges the limitations of RALMs, particularly in retrieval quality and computational efficiency, offering directions for future research. In conclusion, this survey aims to offer a structured insight into RALMs, their potential, and the avenues for their future development in NLP. The paper is supplemented with a Github Repository containing the surveyed works and resources for further study: https://github.com/2471023025/RALM_Survey.
Paper Structure (61 sections, 5 equations, 7 figures, 3 tables)

This paper contains 61 sections, 5 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Definition
  3. Sequential Single Interaction
  4. Sequential Multiple Interactions
  5. Parallel Interaction
  6. Retriever
  7. Sparse Retriever
  8. Word Frequency
  9. Sparse Vector Representation
  10. Dense Retriever
  11. Word Embedding
  12. Multimodal Retrieval
  13. Knowledge Distillation
  14. Internet Retrieval
  15. Hybrid Retrieval
  16. ...and 46 more sections

Figures (7)

  • Figure 1: A general overview of this survey‘s work
  • Figure 2: Three different ways the Retriever interacts with the LM
  • Figure 3: A roadmap of the three types of interactions. The purple areas represent work on Sequential Interaction RALM models, the red boxes signify work on Sequential Multiple Interactions RALMs models, and the yellow areas indicate work on Parallel Interaction RALM models.
  • Figure 4: Classification of RALM enhancement methods.
  • Figure 5: Classification of RALM data sources.
  • ...and 2 more figures