Enhancing Multi-Hop Fact Verification with Structured Knowledge-Augmented Large Language Models

TL;DR

This work tackles multi-hop fact verification in the face of misinformation by marrying structured knowledge with graph-based reasoning. It introduces LLM-SKAN, a framework that uses a fine-tuned LLM as a knowledge extractor to obtain entity-relation-entity triplets and fuses them with claim/evidence data through a Knowledge-Augmented Relation Graph. A graph neural network then learns rich representations for accurate verdict prediction, with experiments on FEVER and HOVER showing state-of-the-art performance and strong ablation results confirming the value of each component. The approach demonstrates that leveraging fine-grained, structured knowledge extracted by LLMs and integrated via graph fusion significantly enhances multi-hop reasoning and verification accuracy, offering practical benefits for misinformation mitigation and robust evidence-based QA systems.

Abstract

The rapid development of social platforms exacerbates the dissemination of misinformation, which stimulates the research in fact verification. Recent studies tend to leverage semantic features to solve this problem as a single-hop task. However, the process of verifying a claim requires several pieces of evidence with complicated inner logic and relations to verify the given claim in real-world situations. Recent studies attempt to improve both understanding and reasoning abilities to enhance the performance, but they overlook the crucial relations between entities that benefit models to understand better and facilitate the prediction. To emphasize the significance of relations, we resort to Large Language Models (LLMs) considering their excellent understanding ability. Instead of other methods using LLMs as the predictor, we take them as relation extractors, for they do better in understanding rather than reasoning according to the experimental results. Thus, to solve the challenges above, we propose a novel Structured Knowledge-Augmented LLM-based Network (LLM-SKAN) for multi-hop fact verification. Specifically, we utilize an LLM-driven Knowledge Extractor to capture fine-grained information, including entities and their complicated relations. Besides, we leverage a Knowledge-Augmented Relation Graph Fusion module to interact with each node and learn better claim-evidence representations comprehensively. The experimental results on four common-used datasets demonstrate the effectiveness and superiority of our model.

Figures (3)

• Figure 1: The framework of our proposed LLM-SKAN. This framework consists of four main components: (1) Structured knowledge-augmented LLM, aiming to fine-tune the LLM to capture more accurate fine-grained relations; (2) LLM-driven knowledge extractor, aiming to extract fine-grained knowledge; (3) Knowledge-augmented relation graph fusion, aiming to learn comprehensive representations through augmented graph; (4) Fact verification, utilizing the comprehensive representation to predict the label.
• Figure 2: The comparison of different relation extraction methods. We compare Llama2 to Mistral-7B and Vicuna-7B. Prompt, QA and RE denote the prompt-tuning, fine-tuning based on the question-answering task, and fine-tuning based on the relation extraction task, respectively.
• Figure 3: The comparison of different claim-evidence fusion methods. Concat, Seq-Att, and RGF denote using the simple concatenation, the sequential attention mechanism, and the relation graph fusion module to model complex relations for verification, respectively.