Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evidence-backed Fact Checking using RAG and Few-Shot In-Context Learning with LLMs

Ronit Singhal, Pransh Patwa, Parth Patwa, Aman Chadha, Amitava Das

TL;DR

This paper develops a Retrieve and Generate pipeline to extract relevant evidence sentences from a knowledge base, which are then inputted along with the claim into a large language model (LLM) for classification, and evaluates the few-shot In-Context Learning (ICL) capabilities of multiple LLMs.

Abstract

Given the widespread dissemination of misinformation on social media, implementing fact-checking mechanisms for online claims is essential. Manually verifying every claim is very challenging, underscoring the need for an automated fact-checking system. This paper presents our system designed to address this issue. We utilize the Averitec dataset (Schlichtkrull et al., 2023) to assess the performance of our fact-checking system. In addition to veracity prediction, our system provides supporting evidence, which is extracted from the dataset. We develop a Retrieve and Generate (RAG) pipeline to extract relevant evidence sentences from a knowledge base, which are then inputted along with the claim into a large language model (LLM) for classification. We also evaluate the few-shot In-Context Learning (ICL) capabilities of multiple LLMs. Our system achieves an 'Averitec' score of 0.33, which is a 22% absolute improvement over the baseline. Our Code is publicly available on https://github.com/ronit-singhal/evidence-backed-fact-checking-using-rag-and-few-shot-in-context-learning-with-llms.

Evidence-backed Fact Checking using RAG and Few-Shot In-Context Learning with LLMs

TL;DR

This paper develops a Retrieve and Generate pipeline to extract relevant evidence sentences from a knowledge base, which are then inputted along with the claim into a large language model (LLM) for classification, and evaluates the few-shot In-Context Learning (ICL) capabilities of multiple LLMs.

Abstract

Given the widespread dissemination of misinformation on social media, implementing fact-checking mechanisms for online claims is essential. Manually verifying every claim is very challenging, underscoring the need for an automated fact-checking system. This paper presents our system designed to address this issue. We utilize the Averitec dataset (Schlichtkrull et al., 2023) to assess the performance of our fact-checking system. In addition to veracity prediction, our system provides supporting evidence, which is extracted from the dataset. We develop a Retrieve and Generate (RAG) pipeline to extract relevant evidence sentences from a knowledge base, which are then inputted along with the claim into a large language model (LLM) for classification. We also evaluate the few-shot In-Context Learning (ICL) capabilities of multiple LLMs. Our system achieves an 'Averitec' score of 0.33, which is a 22% absolute improvement over the baseline. Our Code is publicly available on https://github.com/ronit-singhal/evidence-backed-fact-checking-using-rag-and-few-shot-in-context-learning-with-llms.
Paper Structure (14 sections, 6 figures, 4 tables)

This paper contains 14 sections, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Data
  4. Methodology
  5. Document Retrieval Using Dense Embeddings
  6. Evidence Extraction Using LLMs
  7. Few-Shot ICL for Final Classification
  8. Experiments
  9. Evaluation Metrics
  10. Results and Analysis
  11. Class-wise Performance
  12. Conclusion and Future Work
  13. Limitation
  14. Ethical Statement

Figures (6)

  • Figure 1: Overview diagram of our system. First, the claim is used to retrieve the top 3 relevant documents. Next, evidence is extracted from these documents using questions and answers generated by an LLM. Finally, the evidence is used for veracity prediction via few-shot ICL.
  • Figure 2: Word cloud of the claims. We can see that Politics and COVID-19 are common topics in the claims.
  • Figure 3: The prompt used for generating questions. Some manually created correct and incorrect examples are given to guide the LLM.
  • Figure 4: The prompt used for generating answers. This prompt is repeated for each of the top three documents.
  • Figure 5: A prompt similar to the one used for generating the final prediction. The actual prompt has some more instructions which are omitted here in the interest of space. two annotated train examples are provided for the LLM to learn from.
  • ...and 1 more figures