MultiCaption: Detecting disinformation using multilingual visual claims
Rafael Martins Frade, Rrubaa Panchendrarajan, Arkaitz Zubiaga
TL;DR
MultiCaption addresses the gap in multilingual, multimodal misinformation datasets by introducing 11,088 visual-claim pairs across 64 languages and defining what constitutes contradictory visual claims. The authors construct the dataset using multiple labeling strategies, including manual validation, claim-post links, self-expansion, and LLM-based annotation, to generate both contradicting and non-contradicting pairs. They benchmark a range of transformers, NLI models, and multilingual LLMs, showing that the task is more challenging than standard NLI and benefits strongly from task-specific fine-tuning and multilingual training. The work demonstrates that high-performance multilingual fact-checking pipelines can be built without heavy reliance on translation and discusses practical deployment, limitations, and avenues for extending to multimodal settings.
Abstract
Online disinformation poses an escalating threat to society, driven increasingly by the rapid spread of misleading content across both multimedia and multilingual platforms. While automated fact-checking methods have advanced in recent years, their effectiveness remains constrained by the scarcity of datasets that reflect these real-world complexities. To address this gap, we first present MultiCaption, a new dataset specifically designed for detecting contradictions in visual claims. Pairs of claims referring to the same image or video were labeled through multiple strategies to determine whether they contradict each other. The resulting dataset comprises 11,088 visual claims in 64 languages, offering a unique resource for building and evaluating misinformation-detection systems in truly multimodal and multilingual environments. We then provide comprehensive experiments using transformer-based architectures, natural language inference models, and large language models, establishing strong baselines for future research. The results show that MultiCaption is more challenging than standard NLI tasks, requiring task-specific finetuning for strong performance. Moreover, the gains from multilingual training and testing highlight the dataset's potential for building effective multilingual fact-checking pipelines without relying on machine translation.