CIEC: Coupling Implicit and Explicit Cues for Multimodal Weakly Supervised Manipulation Localization

TL;DR

This work tackles image–text multimodal manipulation localization under weak supervision, removing the need for fine-grained patch/token annotations. It introduces CIEC, a three-component framework that couples implicit unimodal cues with explicit cross-modal guidance via Cross-modal Feature Alignment (CFA), Textual-guidance Refine Patch Selection (TRPS), and Visual-deviation Calibrated Token Grounding (VCTG). TRPS and VCTG are augmented with targeted constraints—background silencing, spatial contrast enhancement, asymmetric sparse activation, and semantic consistency—to robustly localize manipulated regions using only coarse labels. On the DGM^4 benchmark, CIEC achieves strong results that are competitive with fully supervised methods, particularly excelling in image grounding, and demonstrates the effectiveness of combining implicit and explicit cues for multimodal weak supervision.

Abstract

To mitigate the threat of misinformation, multimodal manipulation localization has garnered growing attention. Consider that current methods rely on costly and time-consuming fine-grained annotations, such as patch/token-level annotations. This paper proposes a novel framework named Coupling Implicit and Explicit Cues (CIEC), which aims to achieve multimodal weakly-supervised manipulation localization for image-text pairs utilizing only coarse-grained image/sentence-level annotations. It comprises two branches, image-based and text-based weakly-supervised localization. For the former, we devise the Textual-guidance Refine Patch Selection (TRPS) module. It integrates forgery cues from both visual and textual perspectives to lock onto suspicious regions aided by spatial priors. Followed by the background silencing and spatial contrast constraints to suppress interference from irrelevant areas. For the latter, we devise the Visual-deviation Calibrated Token Grounding (VCTG) module. It focuses on meaningful content words and leverages relative visual bias to assist token localization. Followed by the asymmetric sparse and semantic consistency constraints to mitigate label noise and ensure reliability. Extensive experiments demonstrate the effectiveness of our CIEC, yielding results comparable to fully supervised methods on several evaluation metrics.

Figures (3)

• Figure 1: Comparison of different multimodal manipulation localization paradigms. (a) Fully supervised localization relies on expensive fine-grained annotations. (b) Existing multimodal weakly-supervised forgery localization (audio-visual) only considers scenarios where both modalities are true or fake, benefiting from high cross-modal consistency. (c) Our task (image-text) encounters the dual challenge of cross-modal inconsistency forgery (e.g., TF, FT) and stronger modality independence.
• Figure 2: The overall architecture of proposed CIEC. It consists of three components: (a) Cross-modal Feature Alignment, (b) Textual-guidance Refine Patch Selection and (c) Visual-deviation Calibrated Token Grounding.
• Figure 3: Visualization of detection and grounding results. Here, red box and text indicate the prediction of manipulated faces and words, while green box and text represent the corresponding ground truth.