A Generative-AI-Driven Claim Retrieval System Capable of Detecting and Retrieving Claims from Social Media Platforms in Multiple Languages

TL;DR

The paper tackles the challenge of multilingual online misinformation by proposing a generative-AI driven claim retrieval system that identifies previously fact-checked claims relevant to a given input and supports fact-checkers with concise summaries and explanations. The authors introduce a four-step pipeline—retrieval, filtration, summarization, and veracity prediction—leveraging multilingual TEMs and a suite of LLMs, evaluated on the MultiClaim and AFP-Sum datasets across 10+ languages, with both automatic metrics and human evaluation. Key contributions include the AFP-Sum dataset, a multilingual claim retrieval framework, and a web-based tool integrating retrieval, summarization, and veracity prediction, demonstrated to improve efficiency while highlighting language-dependent performance and the trade-offs of LLM-based filtration. The work shows that multilingual TEMs often outperform English baselines, larger LLMs yield stronger summarization and veracity capabilities, and that careful handling of bias, ethics, and data usage is essential for real-world deployment in cross-language fact-checking contexts.

Abstract

Online disinformation poses a global challenge, placing significant demands on fact-checkers who must verify claims efficiently to prevent the spread of false information. A major issue in this process is the redundant verification of already fact-checked claims, which increases workload and delays responses to newly emerging claims. This research introduces an approach that retrieves previously fact-checked claims, evaluates their relevance to a given input, and provides supplementary information to support fact-checkers. Our method employs large language models (LLMs) to filter irrelevant fact-checks and generate concise summaries and explanations, enabling fact-checkers to faster assess whether a claim has been verified before. In addition, we evaluate our approach through both automatic and human assessments, where humans interact with the developed tool to review its effectiveness. Our results demonstrate that LLMs are able to filter out many irrelevant fact-checks and, therefore, reduce effort and streamline the fact-checking process.

Figures (11)

• Figure 1: An example of a post with two fact-checked claims retrieved by the embedding model. The LLM selects the relevant claim, explains its choice, summarizes the fact-check article, and predicts the post's veracity.
• Figure 2: Our proposed pipeline consisting of (1) retrieval of the top N most similar fact-checks, (2) identifying relevant fact-checked claims, (3) summarizing relevant fact-checking articles, and (4) predicting the veracity of the query along with the explanation.
• Figure 3: Overall performance of LLMs for fact-check summarization. We report the average ROUGE-L score for each LLM using the Article first setup, where the article is provided before the instruction.
• Figure 4: Number of participants ($N=6$) who highlighted each evaluation criterion as beneficial.
• Figure 5: Template used to structure fact-checks for filtered retrieval, along with an example illustrating its format, including the fact-checked claim, language, publication date and fact-checking organization.