Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Retrieval-Augmented Generation by Evidence Retroactivity in LLMs

Liang Xiao, Wen Dai, Shuai Chen, Bin Qin, Chongyang Shi, Haopeng Jing, Tianyu Guo

TL;DR

This work addresses hallucination in large language models by proposing RetroRAG, a retroactive retrieval-augmented generation framework. It introduces the Evidence-collation-and-discovery loop (ELLERY) and an Answerer component that iteratively searches for, generates, and refines evidence to correct reasoning. The approach demonstrates state-of-the-art results on two multi-hop QA datasets (HotpotQA and 2WikiMQA) and provides extensive ablations and qualitative analyses to show the value of evidence updating and retroactive reasoning for reducing external hallucinations. The method achieves these results without fine-tuning or pre-training the LLM, with potential for broader applicability and improved explainability through explicit evidence tracking.

Abstract

Retrieval-augmented generation has gained significant attention due to its ability to integrate relevant external knowledge, enhancing the accuracy and reliability of the LLMs' responses. Most of the existing methods apply a dynamic multiple retrieval-generating process, to address multi-hop complex questions by decomposing them into sub-problems. However, these methods rely on an unidirectional forward reasoning paradigm, where errors from insufficient reasoning steps or inherent flaws in current retrieval systems are irreversible, potentially derailing the entire reasoning chain. For the first time, this work introduces Retroactive Retrieval-Augmented Generation (RetroRAG), a novel framework to build a retroactive reasoning paradigm. RetroRAG revises and updates the evidence, redirecting the reasoning chain to the correct direction. RetroRAG constructs an evidence-collation-discovery framework to search, generate, and refine credible evidence. It synthesizes inferential evidence related to the key entities in the question from the existing source knowledge and formulates search queries to uncover additional information. As new evidence is found, RetroRAG continually updates and organizes this information, enhancing its ability to locate further necessary evidence. Paired with an Answerer to generate and evaluate outputs, RetroRAG is capable of refining its reasoning process iteratively until a reliable answer is obtained. Empirical evaluations show that RetroRAG significantly outperforms existing methods.

Retrieval-Augmented Generation by Evidence Retroactivity in LLMs

TL;DR

This work addresses hallucination in large language models by proposing RetroRAG, a retroactive retrieval-augmented generation framework. It introduces the Evidence-collation-and-discovery loop (ELLERY) and an Answerer component that iteratively searches for, generates, and refines evidence to correct reasoning. The approach demonstrates state-of-the-art results on two multi-hop QA datasets (HotpotQA and 2WikiMQA) and provides extensive ablations and qualitative analyses to show the value of evidence updating and retroactive reasoning for reducing external hallucinations. The method achieves these results without fine-tuning or pre-training the LLM, with potential for broader applicability and improved explainability through explicit evidence tracking.

Abstract

Retrieval-augmented generation has gained significant attention due to its ability to integrate relevant external knowledge, enhancing the accuracy and reliability of the LLMs' responses. Most of the existing methods apply a dynamic multiple retrieval-generating process, to address multi-hop complex questions by decomposing them into sub-problems. However, these methods rely on an unidirectional forward reasoning paradigm, where errors from insufficient reasoning steps or inherent flaws in current retrieval systems are irreversible, potentially derailing the entire reasoning chain. For the first time, this work introduces Retroactive Retrieval-Augmented Generation (RetroRAG), a novel framework to build a retroactive reasoning paradigm. RetroRAG revises and updates the evidence, redirecting the reasoning chain to the correct direction. RetroRAG constructs an evidence-collation-discovery framework to search, generate, and refine credible evidence. It synthesizes inferential evidence related to the key entities in the question from the existing source knowledge and formulates search queries to uncover additional information. As new evidence is found, RetroRAG continually updates and organizes this information, enhancing its ability to locate further necessary evidence. Paired with an Answerer to generate and evaluate outputs, RetroRAG is capable of refining its reasoning process iteratively until a reliable answer is obtained. Empirical evaluations show that RetroRAG significantly outperforms existing methods.
Paper Structure (31 sections, 5 equations, 14 figures, 3 tables)

This paper contains 31 sections, 5 equations, 14 figures, 3 tables.

Table of Contents

  1. Introduction
  2. RELATED WORK
  3. Hallucination in Large Language Model
  4. Retrieval Augmented Language Model
  5. Methodology
  6. Overview
  7. Answerer
  8. Evidence Collation and Discovery
  9. Evidence Collation
  10. Evidence Discovery
  11. Experiments
  12. Experimental Setup
  13. Datasets and Evaluation Metrics
  14. Baselines
  15. Settings
  16. ...and 16 more sections

Figures (14)

  • Figure 1: An example of previous RAG approaches causes hallucinatory content due to their unidirectional forwards reasoning paradigm, and how RetroRAG address this issue.
  • Figure 2: Comparison between previous methods and RetroRAG in mechanism.
  • Figure 3: Overview of our RetroRAG structure.
  • Figure 4: Comparison with different maximum numbers of iterations, numbers of evidence, and similarity thresholds.
  • Figure 5: Case study of RetroRAG and previous approaches.
  • ...and 9 more figures