Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving the fact-checking performance of language models by relying on their entailment ability

Gaurav Kumar, Debajyoti Mazumder, Ayush Garg, Jasabanta Patro

TL;DR

This work introduces an entailment-based fact-checking framework that leverages entailed justifications generated by GLMs to train encoder-only language models for veracity prediction. By organizing the process into three steps—entailment classification of evidence, generation of supporting/refuting justifications, and conditional veracity prediction using an ELM—the approach achieves substantial improvements over strong baselines, especially when training with entailed justifications. Through three training-based experiments and four inference-based experiments on LIAR-RAW and RAW-FC datasets, the authors demonstrate that entailed explanations can greatly enhance accuracy and interpretability, while ablations and linguistic analyses illuminate the critical role of justification content and attention to evidence. The findings suggest practical implications for deploying scalable, explainable fact-checking systems, with future work pointing toward multilingual generalization and open-domain evidence retrieval.

Abstract

Automated fact-checking has been a challenging task for the research community. Past works tried various strategies, such as end-to-end training, retrieval-augmented generation, and prompt engineering, to build robust fact-checking systems. However, their accuracy has not been very high for real-world deployment. We, on the other hand, propose a simple yet effective strategy, where entailed justifications generated by LLMs are used to train encoder-only language models (ELMs) for fact-checking. We conducted a rigorous set of experiments, comparing our approach with recent works and various prompting and fine-tuning strategies to demonstrate the superiority of our approach. Additionally, we did quality analysis of model explanations, ablation studies, and error analysis to provide a comprehensive understanding of our approach.

Improving the fact-checking performance of language models by relying on their entailment ability

TL;DR

This work introduces an entailment-based fact-checking framework that leverages entailed justifications generated by GLMs to train encoder-only language models for veracity prediction. By organizing the process into three steps—entailment classification of evidence, generation of supporting/refuting justifications, and conditional veracity prediction using an ELM—the approach achieves substantial improvements over strong baselines, especially when training with entailed justifications. Through three training-based experiments and four inference-based experiments on LIAR-RAW and RAW-FC datasets, the authors demonstrate that entailed explanations can greatly enhance accuracy and interpretability, while ablations and linguistic analyses illuminate the critical role of justification content and attention to evidence. The findings suggest practical implications for deploying scalable, explainable fact-checking systems, with future work pointing toward multilingual generalization and open-domain evidence retrieval.

Abstract

Automated fact-checking has been a challenging task for the research community. Past works tried various strategies, such as end-to-end training, retrieval-augmented generation, and prompt engineering, to build robust fact-checking systems. However, their accuracy has not been very high for real-world deployment. We, on the other hand, propose a simple yet effective strategy, where entailed justifications generated by LLMs are used to train encoder-only language models (ELMs) for fact-checking. We conducted a rigorous set of experiments, comparing our approach with recent works and various prompting and fine-tuning strategies to demonstrate the superiority of our approach. Additionally, we did quality analysis of model explanations, ablation studies, and error analysis to provide a comprehensive understanding of our approach.

Paper Structure

This paper contains 21 sections, 2 figures, 13 tables.

Table of Contents

  1. Introduction:
  2. Related works:
  3. Dataset details:
  4. Experiments:
  5. Training Based Experiments (TBEs):
  6. TBE-1: Training based on raw-evidences:
  7. TBE-2: Training based on overall understanding:
  8. TBE-3: Training based on entailment understanding:
  9. Inference Based Experiments (IBEs):
  10. Baselines:
  11. Experimental set-up:
  12. Evaluation metrics:
  13. Hyperparameter details:
  14. Results and Discussion:
  15. Observations for veracity prediction:
  16. ...and 6 more sections

Figures (2)

  • Figure 1: A schematic diagram explaining fact-checking based on entailment, which breaks down into three stages. (a) Entailment: Given a claim and its corresponding evidences, a GLM first distinguishes evidences into supporting/ refuting evidence via entailment. (b) Evidence consolidation: Then the same GLM consolidates these two groups into concise supporting and refuting justification. (c) Veracity prediction: Using the claim and both justifications, an ELM is trained to predict veracity.
  • Figure 2: Illustration of steps we followed in different experiments. Sub-figure (a) presents the case where only raw evidence sentences and claims are given as input (R1). This approach is used in TBE-1 and IBE-1. Sub-figure (b) shows the overall process of TBE-2, IBE-2 and IBE-3 based on R2. Sub-figure (c) illustrates the overall experimental process of TBE-3 and IBE-4 based on R3.