Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Robustness of Retrieval-Augmented Language Models with In-Context Learning

Seong-Il Park, Seung-Woo Choi, Na-Hyun Kim, Jay-Yoon Lee

TL;DR

This paper tackles robustness gaps in retrieval-augmented language models (RALMs) for open-domain QA when retrieved contexts are unhelpful or conflicting. It introduces in-context learning with Machine Reading Comprehension demonstrations (cases) that are retrieved and prepended to the LLM input, enabling case-based reasoning without fine-tuning. Across Natural Questions and Web Questions, the approach improves unanswerability and conflict detection accuracy across multiple LLMs, with case selection based on similarity outperforming random selection. Results show that simple demonstrations can significantly bolster robustness, though gains are model-dependent and conflict detection can be challenging for some models such as ChatGPT due to higher false conflict rates.

Abstract

Retrieval-Augmented Language Models (RALMs) have significantly improved performance in open-domain question answering (QA) by leveraging external knowledge. However, RALMs still struggle with unanswerable queries, where the retrieved contexts do not contain the correct answer, and with conflicting information, where different sources provide contradictory answers due to imperfect retrieval. This study introduces an in-context learning-based approach to enhance the reasoning capabilities of RALMs, making them more robust in imperfect retrieval scenarios. Our method incorporates Machine Reading Comprehension (MRC) demonstrations, referred to as cases, to boost the model's capabilities to identify unanswerabilities and conflicts among the retrieved contexts. Experiments on two open-domain QA datasets show that our approach increases accuracy in identifying unanswerable and conflicting scenarios without requiring additional fine-tuning. This work demonstrates that in-context learning can effectively enhance the robustness of RALMs in open-domain QA tasks.

Enhancing Robustness of Retrieval-Augmented Language Models with In-Context Learning

TL;DR

This paper tackles robustness gaps in retrieval-augmented language models (RALMs) for open-domain QA when retrieved contexts are unhelpful or conflicting. It introduces in-context learning with Machine Reading Comprehension demonstrations (cases) that are retrieved and prepended to the LLM input, enabling case-based reasoning without fine-tuning. Across Natural Questions and Web Questions, the approach improves unanswerability and conflict detection accuracy across multiple LLMs, with case selection based on similarity outperforming random selection. Results show that simple demonstrations can significantly bolster robustness, though gains are model-dependent and conflict detection can be challenging for some models such as ChatGPT due to higher false conflict rates.

Abstract

Retrieval-Augmented Language Models (RALMs) have significantly improved performance in open-domain question answering (QA) by leveraging external knowledge. However, RALMs still struggle with unanswerable queries, where the retrieved contexts do not contain the correct answer, and with conflicting information, where different sources provide contradictory answers due to imperfect retrieval. This study introduces an in-context learning-based approach to enhance the reasoning capabilities of RALMs, making them more robust in imperfect retrieval scenarios. Our method incorporates Machine Reading Comprehension (MRC) demonstrations, referred to as cases, to boost the model's capabilities to identify unanswerabilities and conflicts among the retrieved contexts. Experiments on two open-domain QA datasets show that our approach increases accuracy in identifying unanswerable and conflicting scenarios without requiring additional fine-tuning. This work demonstrates that in-context learning can effectively enhance the robustness of RALMs in open-domain QA tasks.
Paper Structure (22 sections, 2 figures, 5 tables)

This paper contains 22 sections, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. In-context learning and RALMs
  4. Robustness of RALMs on unanswerability
  5. Robustness of RALMs on conflicts
  6. Method
  7. Crafting cases
  8. Case retrieval
  9. Experimental Setup
  10. Dataset
  11. Prompting
  12. Metric
  13. LLM
  14. Experiments
  15. Experiments on Unanswerable Set
  16. ...and 7 more sections

Figures (2)

  • Figure 1: Examples of unanswerable and conflict scenario that may arise during retrieval-augmenation. A robust RALM should be able to identify such scenarios well.
  • Figure 2: An overview of our approach. Conventional RALM generates answers by providing the LLM with context retrieved from a knowledge source. In contrast, our method simultaneously inputs cases that enhance the LLM's reasoning capability, allowing it to generate answers. This leads to more robust reasoning compared to conventional RALM.