Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Navigating the Noisy Crowd: Finding Key Information for Claim Verification

Haisong Gong, Huanhuan Ma, Qiang Liu, Shu Wu, Liang Wang

TL;DR

The paper tackles claim verification under noisy evidence by introducing EACon, a framework that first abstracts evidence through keyword-guided extraction and summarization, then deconstructs claims into subclaims for independent verification. It combines Evidence Abstraction, Claim Deconstruction, and Subclaim Verification to reduce noise and improve LLM-based verification, using fuzzy keyword matching and selective summarization to guide evidence processing. Empirical results on HOVER and FEVEROUS-S with Vicuna-13B and Mixtral-8x7B show consistent Macro-F1 gains and robust ablation evidence for the two core components, with larger gains on more complex, multi-hop tasks. The work demonstrates a practical strategy to leverage LLMs for reliable claim verification in noisy real-world information environments, offering a scalable approach that can generalize across datasets and backbones.

Abstract

Claim verification is a task that involves assessing the truthfulness of a given claim based on multiple evidence pieces. Using large language models (LLMs) for claim verification is a promising way. However, simply feeding all the evidence pieces to an LLM and asking if the claim is factual does not yield good results. The challenge lies in the noisy nature of both the evidence and the claim: evidence passages typically contain irrelevant information, with the key facts hidden within the context, while claims often convey multiple aspects simultaneously. To navigate this "noisy crowd" of information, we propose EACon (Evidence Abstraction and Claim Deconstruction), a framework designed to find key information within evidence and verify each aspect of a claim separately. EACon first finds keywords from the claim and employs fuzzy matching to select relevant keywords for each raw evidence piece. These keywords serve as a guide to extract and summarize critical information into abstracted evidence. Subsequently, EACon deconstructs the original claim into subclaims, which are then verified against both abstracted and raw evidence individually. We evaluate EACon using two open-source LLMs on two challenging datasets. Results demonstrate that EACon consistently and substantially improve LLMs' performance in claim verification.

Navigating the Noisy Crowd: Finding Key Information for Claim Verification

TL;DR

The paper tackles claim verification under noisy evidence by introducing EACon, a framework that first abstracts evidence through keyword-guided extraction and summarization, then deconstructs claims into subclaims for independent verification. It combines Evidence Abstraction, Claim Deconstruction, and Subclaim Verification to reduce noise and improve LLM-based verification, using fuzzy keyword matching and selective summarization to guide evidence processing. Empirical results on HOVER and FEVEROUS-S with Vicuna-13B and Mixtral-8x7B show consistent Macro-F1 gains and robust ablation evidence for the two core components, with larger gains on more complex, multi-hop tasks. The work demonstrates a practical strategy to leverage LLMs for reliable claim verification in noisy real-world information environments, offering a scalable approach that can generalize across datasets and backbones.

Abstract

Claim verification is a task that involves assessing the truthfulness of a given claim based on multiple evidence pieces. Using large language models (LLMs) for claim verification is a promising way. However, simply feeding all the evidence pieces to an LLM and asking if the claim is factual does not yield good results. The challenge lies in the noisy nature of both the evidence and the claim: evidence passages typically contain irrelevant information, with the key facts hidden within the context, while claims often convey multiple aspects simultaneously. To navigate this "noisy crowd" of information, we propose EACon (Evidence Abstraction and Claim Deconstruction), a framework designed to find key information within evidence and verify each aspect of a claim separately. EACon first finds keywords from the claim and employs fuzzy matching to select relevant keywords for each raw evidence piece. These keywords serve as a guide to extract and summarize critical information into abstracted evidence. Subsequently, EACon deconstructs the original claim into subclaims, which are then verified against both abstracted and raw evidence individually. We evaluate EACon using two open-source LLMs on two challenging datasets. Results demonstrate that EACon consistently and substantially improve LLMs' performance in claim verification.
Paper Structure (23 sections, 6 equations, 2 figures, 4 tables)

This paper contains 23 sections, 6 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Claim Verification
  4. Large Language Model Reasoning
  5. Method
  6. Task Formulation
  7. Evidence Abstraction
  8. Keyword Extraction
  9. Keyword Selection
  10. Evidence Summarization
  11. Claim Deconstruction
  12. Subclaim Verification
  13. Experiment
  14. Dataset
  15. Baselines
  16. ...and 8 more sections

Figures (2)

  • Figure 1: Architecture of EACon. The input is a claim and raw evidence, and the output is the predicted veracity of the claim. EACon extracts keywords from the claim and uses fuzzy matching to select keywords for each piece of the raw evidence. These selected keywords are then used to summarize the raw evidence into abstracted evidence. EACon then deconstructs the claim into subclaims, which are verified against both the raw and abstracted evidence using a zero-shot approach.
  • Figure 2: Illustration of different methods for prompting LLM to abstract evidence. From left to right: (I) Prompting LLM based on claim leads to incorrect output due to conflicting claim and evidence. (II) Prompting LLM with all keywords may result in generating content not supported by the evidence. (III) Our proposed method using selected keywords leads to correct output.