FIRE: Fact-checking with Iterative Retrieval and Verification

TL;DR

FIRE proposes an iterative, agent-based framework that unifies evidence retrieval and claim verification, enabling LLMs to use their internal knowledge first and fetch external evidence only when needed. By integrating search and verification into a single decision process, FIRE reduces both LLM computation costs and web search usage while maintaining competitive factuality across multiple datasets. Ablation studies show that step-by-step reasoning can increase confidence and reduce unnecessary searches, though aggressive evidence retrieval may not always improve performance. The work highlights dataset quality issues and outlines future directions including memory integration and multimodal extensions to broaden applicability and robustness.

Abstract

Fact-checking long-form text is challenging, and it is therefore common practice to break it down into multiple atomic claims. The typical approach to fact-checking these atomic claims involves retrieving a fixed number of pieces of evidence, followed by a verification step. However, this method is usually not cost-effective, as it underutilizes the verification model's internal knowledge of the claim and fails to replicate the iterative reasoning process in human search strategies. To address these limitations, we propose FIRE, a novel agent-based framework that integrates evidence retrieval and claim verification in an iterative manner. Specifically, FIRE employs a unified mechanism to decide whether to provide a final answer or generate a subsequent search query, based on its confidence in the current judgment. We compare FIRE with other strong fact-checking frameworks and find that it achieves slightly better performance while reducing large language model (LLM) costs by an average of 7.6 times and search costs by 16.5 times. These results indicate that FIRE holds promise for application in large-scale fact-checking operations. Our code is available at https://github.com/mbzuai-nlp/fire.git.

Figures (4)

• Figure 1: Comparisons between Fire and previous frameworks. Previous frameworks typically treat web search and claim verification as distinct processes. In contrast, Fire integrates interactive retrieval and verification.
• Figure 2: The effect of reasoning on the number of searches using GPT-4o and GPT-4o-mini within Fire on BingCheck. The shaded area indicates the number of misclassified cases. The x-axis shows the number of web searches, while the y-axis denotes the number of instances.
• Figure 3: The effect of reasoning on the number of searches using GPT-4o and GPT-4o-mini within Fire on FacTool-QA.
• Figure 4: The effect of reasoning on the number of searches using GPT-4o and GPT-4o-mini within Fire on FELM-WK.