Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reliable, Adaptable, and Attributable Language Models with Retrieval

Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hannaneh Hajishirzi, Wen-tau Yih

TL;DR

Parametric LMs memorize vast web data but suffer from hallucinations, verification challenges, data-provenance concerns, costly adaptation, and unwieldy size. The authors advocate retrieval-augmented LMs that query external datastores at inference to enhance reliability, adaptability, and attribution, arguing they can outperform purely parametric systems across broad tasks. The paper surveys architectures, training regimes, and datastores, identifies adoption barriers, and outlines a roadmap spanning datastore redesign, deeper retriever-LM integration, and scalable infrastructure. It concludes by calling for cross-disciplinary collaboration and open-source tooling to scale retrieval-augmented LMs for real-world deployment.

Abstract

Parametric language models (LMs), which are trained on vast amounts of web data, exhibit remarkable flexibility and capability. However, they still face practical challenges such as hallucinations, difficulty in adapting to new data distributions, and a lack of verifiability. In this position paper, we advocate for retrieval-augmented LMs to replace parametric LMs as the next generation of LMs. By incorporating large-scale datastores during inference, retrieval-augmented LMs can be more reliable, adaptable, and attributable. Despite their potential, retrieval-augmented LMs have yet to be widely adopted due to several obstacles: specifically, current retrieval-augmented LMs struggle to leverage helpful text beyond knowledge-intensive tasks such as question answering, have limited interaction between retrieval and LM components, and lack the infrastructure for scaling. To address these, we propose a roadmap for developing general-purpose retrieval-augmented LMs. This involves a reconsideration of datastores and retrievers, the exploration of pipelines with improved retriever-LM interaction, and significant investment in infrastructure for efficient training and inference.

Reliable, Adaptable, and Attributable Language Models with Retrieval

TL;DR

Parametric LMs memorize vast web data but suffer from hallucinations, verification challenges, data-provenance concerns, costly adaptation, and unwieldy size. The authors advocate retrieval-augmented LMs that query external datastores at inference to enhance reliability, adaptability, and attribution, arguing they can outperform purely parametric systems across broad tasks. The paper surveys architectures, training regimes, and datastores, identifies adoption barriers, and outlines a roadmap spanning datastore redesign, deeper retriever-LM integration, and scalable infrastructure. It concludes by calling for cross-disciplinary collaboration and open-source tooling to scale retrieval-augmented LMs for real-world deployment.

Abstract

Parametric language models (LMs), which are trained on vast amounts of web data, exhibit remarkable flexibility and capability. However, they still face practical challenges such as hallucinations, difficulty in adapting to new data distributions, and a lack of verifiability. In this position paper, we advocate for retrieval-augmented LMs to replace parametric LMs as the next generation of LMs. By incorporating large-scale datastores during inference, retrieval-augmented LMs can be more reliable, adaptable, and attributable. Despite their potential, retrieval-augmented LMs have yet to be widely adopted due to several obstacles: specifically, current retrieval-augmented LMs struggle to leverage helpful text beyond knowledge-intensive tasks such as question answering, have limited interaction between retrieval and LM components, and lack the infrastructure for scaling. To address these, we propose a roadmap for developing general-purpose retrieval-augmented LMs. This involves a reconsideration of datastores and retrievers, the exploration of pipelines with improved retriever-LM interaction, and significant investment in infrastructure for efficient training and inference.
Paper Structure (48 sections, 2 figures, 2 tables)

This paper contains 48 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. How Far Can We Go with Parametric LMs?
  3. Definition.
  4. Weaknesses of Parametric LMs
  5. W2: Difficulty of verifications.
  6. W3: Difficulty of opting out certain sequences from the datasets.
  7. W4: Computationally expensive costs to adapt.
  8. W5: Prohibitively large model size.
  9. How Can Retrieval-Augmented LMs Address These Issues?
  10. Effectiveness of Retrieval-Augmented LMs
  11. W1: Reduced factual errors in long-tail knowledge.
  12. W2: Better attributions.
  13. W3: Enabling flexible opt-in of sequences.
  14. W4: Adaptability and customizability.
  15. W5: Parameter efficiency.
  16. ...and 33 more sections

Figures (2)

  • Figure 1: Parametric LMs (top) internalize large-scale text data in their parameters via massive pre-training, while retrieval-augmented LMs (bottom) incorporate text retrieved from a massive datastore at test time.
  • Figure 2: Taxonomy of architectures of retrieval-augmented LMs.