Verify-in-the-Graph: Entity Disambiguation Enhancement for Complex Claim Verification with Interactive Graph Representation

TL;DR

VeGraph introduces an interactive graph-based framework where an LLM agent converts complex claims into graph triplets, iteratively disambiguates ambiguous entities via KB interactions, and verifies remaining sub-claims to determine veracity. The approach enhances explainability by logging reasoning steps and supports multi-hop reasoning through iterative graph refinement. Evaluations on HoVer and FEVEROUS with Meta-Llama-3-Instruct show competitive performance, with notable gains in complex four-hop claims and disambiguation tasks, albeit with higher computational costs. The work highlights the practical potential of integrating structured graph representations with agent-based reasoning for robust, interpretable claim verification in open-book settings.

Abstract

Claim verification is a long-standing and challenging task that demands not only high accuracy but also explainability of the verification process. This task becomes an emerging research issue in the era of large language models (LLMs) since real-world claims are often complex, featuring intricate semantic structures or obfuscated entities. Traditional approaches typically address this by decomposing claims into sub-claims and querying a knowledge base to resolve hidden or ambiguous entities. However, the absence of effective disambiguation strategies for these entities can compromise the entire verification process. To address these challenges, we propose Verify-in-the-Graph (VeGraph), a novel framework leveraging the reasoning and comprehension abilities of LLM agents. VeGraph operates in three phases: (1) Graph Representation - an input claim is decomposed into structured triplets, forming a graph-based representation that integrates both structured and unstructured information; (2) Entity Disambiguation -VeGraph iteratively interacts with the knowledge base to resolve ambiguous entities within the graph for deeper sub-claim verification; and (3) Verification - remaining triplets are verified to complete the fact-checking process. Experiments using Meta-Llama-3-70B (instruct version) show that VeGraph achieves competitive performance compared to baselines on two benchmarks HoVer and FEVEROUS, effectively addressing claim verification challenges. Our source code and data are available for further exploitation.

Figures (15)

• Figure 1: Conceptual analysis of previous works and VeGraph: a) Traditional approaches use IR to retrieve evidence and then verify sub-claims; b) Advanced approaches use IR to resolve ambiguous entities and then verify sub-claims; c) Our approach represents claims with graph triplets, then iteratively interacts with IR for entity disambiguation and sub-claims verification.
• Figure 2: Three key components of VeGraph: (i) Graph Representation, which decomposes the complex input claim into graph triplets; (ii) Entity Disambiguation, ambiguous entities are resolved through iterative interactions with the knowledge base (KB); and (iii) Sub-claim Verification, which evaluates each triplet by delegating the verification process to the sub-claim verification function. The logging module records the whole process.
• Figure 3: Prompt to make LLM construct the Graph Representation
• Figure 4: Illustration of the entity disambiguation process
• Figure 5: Incorrect Example 1