LADLE-MM: Limited Annotation based Detector with Learned Ensembles for Multimodal Misinformation

TL;DR

The paper tackles multimodal misinformation detection under limited annotation and resource constraints. It proposes LADLE-MM, a three-branch architecture with a fixed multimodal BLIP reference space and model-soup initialization to create a lightweight yet robust detector. Key contributions include the fixed BLIP reference as a semantic anchor, a six-block cross-attention fusion scheme, and a manipulation-aware contrastive objective with cosine alignment, yielding competitive DGM4 performance and state-of-the-art open-set results on VERITE while using substantially fewer trainable parameters. Ablation studies validate the importance of the fixed reference, fusion depth, and training strategy, and the method demonstrates strong generalization and resistance to unimodal bias, highlighting practical applicability for scalable misinformation detection.

Abstract

With the rise of easily accessible tools for generating and manipulating multimedia content, realistic synthetic alterations to digital media have become a widespread threat, often involving manipulations across multiple modalities simultaneously. Recently, such techniques have been increasingly employed to distort narratives of important events and to spread misinformation on social media, prompting the development of misinformation detectors. In the context of misinformation conveyed through image-text pairs, several detection methods have been proposed. However, these approaches typically rely on computationally intensive architectures or require large amounts of annotated data. In this work we introduce LADLE-MM: Limited Annotation based Detector with Learned Ensembles for Multimodal Misinformation, a model-soup initialized multimodal misinformation detector designed to operate under a limited annotation setup and constrained training resources. LADLE-MM is composed of two unimodal branches and a third multimodal one that enhances image and text representations with additional multimodal embeddings extracted from BLIP, serving as fixed reference space. Despite using 60.3% fewer trainable parameters than previous state-of-the-art models, LADLE-MM achieves competitive performance on both binary and multi-label classification tasks on the DGM4 benchmark, outperforming existing methods when trained without grounding annotations. Moreover, when evaluated on the VERITE dataset, LADLE-MM outperforms current state-of-the-art approaches that utilize more complex architectures involving Large Vision-Language-Models, demonstrating the effective generalization ability in an open-set setting and strong robustness to unimodal bias.

Figures (6)

• Figure 1: Overview of the proposed LADLE-MM: 1) Feature extraction module: ViT for pure image feature extraction; DeBERTa for pure text feature extraction; BLIP for image-text pair feature extraction. 2) Multimodal feature fusion module consisting of six dual-path cross-attention blocks. 3) Binary and multi-label classification heads.
• Figure 2: On the left an original image-text pair retrieved from the VisualNews dataset, on the right the corresponding manipulated sample extracted from DGM4.
• Figure 3: A three-dimensional Principal Component Analysis (PCA) visualization of the classification tokens extracted from the fusion module. The axes represent the three most significant directions of variance in the data: PC1 (capturing 12.54% of the variance), PC2 (8.60%), and PC3 (7.01%).
• Figure 4: Attention map obtained using the beyond attention method on the cross-attention layers of the text-image branch. The highlighted areas show where the model focuses to detect manipulations.
• Figure 5: Confusion matrix for the multi-label classification task. The visualization highlights that misclassifications primarily occur between manipulation classes and the "no alteration" category, rather than confusing distinct manipulation types with each other, demonstrating the model discriminative robustness.