Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Contrastive Learning to Improve Retrieval for Real-world Fact Checking

Aniruddh Sriram, Fangyuan Xu, Eunsol Choi, Greg Durrett

TL;DR

This work fine-tunes Contriever with a contrastive objective based on multiple training signals, including distillation from GPT-4, evaluating subquestion answers, and gold labels in the dataset, and evaluates the model on both retrieval and end-to-end veracity judgments about claims.

Abstract

Recent work on fact-checking addresses a realistic setting where models incorporate evidence retrieved from the web to decide the veracity of claims. A bottleneck in this pipeline is in retrieving relevant evidence: traditional methods may surface documents directly related to a claim, but fact-checking complex claims requires more inferences. For instance, a document about how a vaccine was developed is relevant to addressing claims about what it might contain, even if it does not address them directly. We present Contrastive Fact-Checking Reranker (CFR), an improved retriever for this setting. By leveraging the AVeriTeC dataset, which annotates subquestions for claims with human written answers from evidence documents, we fine-tune Contriever with a contrastive objective based on multiple training signals, including distillation from GPT-4, evaluating subquestion answers, and gold labels in the dataset. We evaluate our model on both retrieval and end-to-end veracity judgments about claims. On the AVeriTeC dataset, we find a 6\% improvement in veracity classification accuracy. We also show our gains can be transferred to FEVER, ClaimDecomp, HotpotQA, and a synthetic dataset requiring retrievers to make inferences.

Contrastive Learning to Improve Retrieval for Real-world Fact Checking

TL;DR

This work fine-tunes Contriever with a contrastive objective based on multiple training signals, including distillation from GPT-4, evaluating subquestion answers, and gold labels in the dataset, and evaluates the model on both retrieval and end-to-end veracity judgments about claims.

Abstract

Recent work on fact-checking addresses a realistic setting where models incorporate evidence retrieved from the web to decide the veracity of claims. A bottleneck in this pipeline is in retrieving relevant evidence: traditional methods may surface documents directly related to a claim, but fact-checking complex claims requires more inferences. For instance, a document about how a vaccine was developed is relevant to addressing claims about what it might contain, even if it does not address them directly. We present Contrastive Fact-Checking Reranker (CFR), an improved retriever for this setting. By leveraging the AVeriTeC dataset, which annotates subquestions for claims with human written answers from evidence documents, we fine-tune Contriever with a contrastive objective based on multiple training signals, including distillation from GPT-4, evaluating subquestion answers, and gold labels in the dataset. We evaluate our model on both retrieval and end-to-end veracity judgments about claims. On the AVeriTeC dataset, we find a 6\% improvement in veracity classification accuracy. We also show our gains can be transferred to FEVER, ClaimDecomp, HotpotQA, and a synthetic dataset requiring retrievers to make inferences.
Paper Structure (54 sections, 1 equation, 2 figures, 10 tables)

This paper contains 54 sections, 1 equation, 2 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Retrieval Augmented Generation Systems
  4. Limitations of Existing Retrieval Systems
  5. Motivating Example: AVeriTeC
  6. Retrieval signals in fact-checking
  7. Context in retrieval
  8. Methodology
  9. Components
  10. Dense retriever $r$
  11. Reader model
  12. Learning
  13. Implementation Details
  14. Generating Contrastive Training Data
  15. AVeriTeC Gold Evidence
  16. ...and 39 more sections

Figures (2)

  • Figure 1: Top-1 retrieved document from base Contriever (red) and CFR (green). Our model is able to choose a better document despite both paragraphs being topical. Our model recognizes the question is asking about the chemical composition of REGN-COV2, while the unfinetuned model selects a relevant document that does not address "fetal tissue" or help with a final veracity judgment.
  • Figure 2: Overview of generating positive and negative examples for finetuning the retriever. We first select documents with high BM25 score with the (query, subquestion) from both the web documents and gold articles. We then experiment with different methods (described in Section \ref{['subsec:training_data_generation']}) to derive positive and hard negative examples.