Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KG-CRAFT: Knowledge Graph-based Contrastive Reasoning with LLMs for Enhancing Automated Fact-checking

Vítor N. Lourenço, Aline Paes, Tillman Weyde, Audrey Depeige, Mohnish Dubey

TL;DR

KG-CRAFT introduces a knowledge graph-based contrastive reasoning framework to enhance automated fact-checking with LLMs. It builds a knowledge graph from a claim and its reports, generates KG-grounded contrastive questions, answers them using the reports, and distills the results into a concise evidence summary for veracity prediction. Across LIAR-RAW and RAWFC, KG-CRAFT achieves new state-of-the-art performance, with KG-CRAFT_L3.3 attaining top F1 scores on both datasets and ablations confirming the value of KG-based contrastive reasoning, especially for smaller models. The approach highlights the benefits of structured reasoning and contrastive explanations for robust, interpretable fact-checking in bounded contexts.

Abstract

Claim verification is a core component of automated fact-checking systems, aimed at determining the truthfulness of a statement by assessing it against reliable evidence sources such as documents or knowledge bases. This work presents KG-CRAFT, a method that improves automatic claim verification by leveraging large language models (LLMs) augmented with contrastive questions grounded in a knowledge graph. KG-CRAFT first constructs a knowledge graph from claims and associated reports, then formulates contextually relevant contrastive questions based on the knowledge graph structure. These questions guide the distillation of evidence-based reports, which are synthesised into a concise summary that is used for veracity assessment by LLMs. Extensive evaluations on two real-world datasets (LIAR-RAW and RAWFC) demonstrate that our method achieves a new state-of-the-art in predictive performance. Comprehensive analyses validate in detail the effectiveness of our knowledge graph-based contrastive reasoning approach in improving LLMs' fact-checking capabilities.

KG-CRAFT: Knowledge Graph-based Contrastive Reasoning with LLMs for Enhancing Automated Fact-checking

TL;DR

KG-CRAFT introduces a knowledge graph-based contrastive reasoning framework to enhance automated fact-checking with LLMs. It builds a knowledge graph from a claim and its reports, generates KG-grounded contrastive questions, answers them using the reports, and distills the results into a concise evidence summary for veracity prediction. Across LIAR-RAW and RAWFC, KG-CRAFT achieves new state-of-the-art performance, with KG-CRAFT_L3.3 attaining top F1 scores on both datasets and ablations confirming the value of KG-based contrastive reasoning, especially for smaller models. The approach highlights the benefits of structured reasoning and contrastive explanations for robust, interpretable fact-checking in bounded contexts.

Abstract

Claim verification is a core component of automated fact-checking systems, aimed at determining the truthfulness of a statement by assessing it against reliable evidence sources such as documents or knowledge bases. This work presents KG-CRAFT, a method that improves automatic claim verification by leveraging large language models (LLMs) augmented with contrastive questions grounded in a knowledge graph. KG-CRAFT first constructs a knowledge graph from claims and associated reports, then formulates contextually relevant contrastive questions based on the knowledge graph structure. These questions guide the distillation of evidence-based reports, which are synthesised into a concise summary that is used for veracity assessment by LLMs. Extensive evaluations on two real-world datasets (LIAR-RAW and RAWFC) demonstrate that our method achieves a new state-of-the-art in predictive performance. Comprehensive analyses validate in detail the effectiveness of our knowledge graph-based contrastive reasoning approach in improving LLMs' fact-checking capabilities.
Paper Structure (44 sections, 5 equations, 3 figures, 7 tables, 1 algorithm)

This paper contains 44 sections, 5 equations, 3 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Knowledge Graph-based Contrastive Reasoning
  4. Knowledge Graph Extraction
  5. Contrastive Reasoning
  6. Verification of Claim Veracity
  7. Experiments and Results
  8. Experimental Settings
  9. Claim Verification Evaluation
  10. Ablation Studies
  11. Impact of Using LLM-generated Contrastive Questions
  12. Impact of the Number of Contrastive Questions
  13. Using Small Language Models
  14. Conclusion
  15. Background
  16. ...and 29 more sections

Figures (3)

  • Figure 1: Overview of the KG-CRAFT framework for automated fact-checking. The method comprises three main phases: (1) knowledge graph extraction from the claim and associated reports, (2) contrastive reasoning, including contrastive question formulation, answer generation, and answer summarisation, and (3) claim veracity prediction.
  • Figure 2: Impact of varying the number of contrastive questions $k$ on fact-checking performance.
  • Figure 3: Performance comparison of Small Language Models incorporated within KG-CRAFT against larger models.