FACT-GPT: Fact-Checking Augmentation via Claim Matching with LLMs

TL;DR

This work investigates augmenting fact-checking with LLMs by framing claim matching as a three-class textual entailment task (Entailment, Neutral, Contradiction). The authors generate synthetic training data from multiple LLMs and create a ground-truth dataset annotated by humans, then compare pre-trained and fine-tuned models, including LoRA-based LLaMA variants. Key findings show that fine-tuned, smaller models can match or approach the performance of larger models, with strongest accuracy on Entailment and Neutral and weaker performance on Contradiction, highlighting the challenge of detecting rebuttals. The study demonstrates the feasibility and value of automated claim matching to support fact-checkers and content moderation while underscoring the need for careful integration and ongoing improvement.

Abstract

Our society is facing rampant misinformation harming public health and trust. To address the societal challenge, we introduce FACT-GPT, a system leveraging Large Language Models (LLMs) to automate the claim matching stage of fact-checking. FACT-GPT, trained on a synthetic dataset, identifies social media content that aligns with, contradicts, or is irrelevant to previously debunked claims. Our evaluation shows that our specialized LLMs can match the accuracy of larger models in identifying related claims, closely mirroring human judgment. This research provides an automated solution for efficient claim matching, demonstrates the potential of LLMs in supporting fact-checkers, and offers valuable resources for further research in the field.

Figures (4)

• Figure 1: Overview of FACT-GPT, our framework aimed at assisting the claim matching stage of the fact-checking process
• Figure 2: Example of synthetic tweet generation prompts
• Figure 3: Workflow of test data construction
• Figure 4: Example of an entailment task prompt