Breaking Language Barriers with MMTweets: Advancing Cross-Lingual Debunked Narrative Retrieval for Fact-Checking

TL;DR

This work addresses cross-lingual debunked narrative retrieval (X-DNR) by introducing the MMTweets benchmark, which pairs multilingual misinformation tweets with debunks across 11 languages and includes images and fine-grained annotations. It benchmarks a range of retrieval models, including translation-based BM25, multilingual dense retrievers (xDPR, mContriever), and various bi-encoder configurations, plus two multistage retrieval pipelines (BE+CE and BE+GPT3.5) that optimize cross-lingual ranking without translation overhead. The study reveals that X-DNR presents notable cross-lingual and cross-dataset transfer challenges, yet multistage retrieval (especially BE+CE) yields the strongest overall ranking performance (e.g., average $nDCG@1$ ≈ 0.728, $nDCG@5$ ≈ 0.669, $MRR$ ≈ 0.795) while highlighting latency trade-offs across methods. The dataset and annotation codebook are publicly available, enabling broader evaluation and iteration to empower multilingual fact-checking workflows and reduce reliance on language-specific resources.

Abstract

Finding previously debunked narratives involves identifying claims that have already undergone fact-checking. The issue intensifies when similar false claims persist in multiple languages, despite the availability of debunks for several months in another language. Hence, automatically finding debunks (or fact-checks) in multiple languages is crucial to make the best use of scarce fact-checkers' resources. Mainly due to the lack of readily available data, this is an understudied problem, particularly when considering the cross-lingual scenario, i.e. the retrieval of debunks in a language different from the language of the online post being checked. This study introduces cross-lingual debunked narrative retrieval and addresses this research gap by: (i) creating Multilingual Misinformation Tweets (MMTweets): a dataset that stands out, featuring cross-lingual pairs, images, human annotations, and fine-grained labels, making it a comprehensive resource compared to its counterparts; (ii) conducting an extensive experiment to benchmark state-of-the-art cross-lingual retrieval models and introducing multistage retrieval methods tailored for the task; and (iii) comprehensively evaluating retrieval models for their cross-lingual and cross-dataset transfer capabilities within MMTweets, and conducting a retrieval latency analysis. We find that MMTweets presents challenges for cross-lingual debunked narrative retrieval, highlighting areas for improvement in retrieval models. Nonetheless, the study provides valuable insights for creating MMTweets datasets and optimising debunked narrative retrieval models to empower fact-checking endeavours. The dataset and annotation codebook are publicly available at https://doi.org/10.5281/zenodo.10637161.

Figures (7)

• Figure 1: Cross-lingual debunked narrative retrieval: Query tweet is in Hindi and the relevant debunk is in English.
• Figure 2: Cross-language analysis: tweet vs. debunk.
• Figure 3: Line plot for month-by-month breakdown of tweet counts for each language in the MMTweets dataset.
• Figure 4: Time gap between tweet and debunk.
• Figure 5: Stacked bar plot for MRR scores for zero-shot cross-lingual transfer and the default results (from Table \ref{['tab:DebRes1']}).