ClaimVer: Explainable Claim-Level Verification and Evidence Attribution of Text Through Knowledge Graphs

TL;DR

ClaimVer tackles the problem of misinformation by providing explainable, claim-level verification against a knowledge graph. It decomposes text into individual claims, retrieves multi-hop KG triplets, and uses fine-tuned LLMs to generate per-claim predictions with rationale and evidence, augmented by a KG Attribution Score ($KAS$) that aggregates evidence via a Triplets Match Score ($TMS$) and a Claim Score ($cs(y_i)$). The approach leverages Wikidata as the KG and a WikiQA-derived dataset with open-source LLMs fine-tuned through LoRA, achieving strong ROUGE and claim-accuracy metrics and delivering actionable per-claim explanations for downstream tasks. This work advances explainable fact-checking by enabling granular attribution, supporting trust, and facilitating downstream applications like ranking and filtering across open-domain texts.

Abstract

In the midst of widespread misinformation and disinformation through social media and the proliferation of AI-generated texts, it has become increasingly difficult for people to validate and trust information they encounter. Many fact-checking approaches and tools have been developed, but they often lack appropriate explainability or granularity to be useful in various contexts. A text validation method that is easy to use, accessible, and can perform fine-grained evidence attribution has become crucial. More importantly, building user trust in such a method requires presenting the rationale behind each prediction, as research shows this significantly influences people's belief in automated systems. Localizing and bringing users' attention to the specific problematic content is also paramount, instead of providing simple blanket labels. In this paper, we present ClaimVer, a human-centric framework tailored to meet users' informational and verification needs by generating rich annotations and thereby reducing cognitive load. Designed to deliver comprehensive evaluations of texts, it highlights each claim, verifies it against a trusted knowledge graph (KG), presents the evidence, and provides succinct, clear explanations for each claim prediction. Finally, our framework introduces an attribution score, enhancing applicability across a wide range of downstream tasks.

Figures (12)

• Figure 1: Demonstration of ClaimVer for claim verification and evidence attribution. (A) Text labeled as Inaccurate by HealthFeedback and ClaimVer's predictions, rationale, and evidence. (B) Text labeled as False by Google Fact Check Tools and ClaimVer's outputs. Predictions are color-coded (amber: extrapolatory, red: contradictory); $R_i$: rationale; related wiki entities are displayed in boxes.
• Figure 2: Flow of operations in the ClaimVer framework. Identified KG entity nodes during preprocessing inform the extraction of relevant triplets by the KG algorithm. Subsequently, these triplets and preprocessed text are then fed to a ClaimVer LLM, fine-tuned to operationalize the objective function. For each claim, the corresponding text span, prediction, relevant triplets, attribution scores, and rationale are generated.
• Figure 3: Instruction prompt for fine-tuned LLMs.
• Figure 4: Fine-tuning loss plots for Llama3-8B-Chat.
• Figure 5: Fine-tuning loss plots for Mistral-7B-v0.3-Chat.