M4-BLIP: Advancing Multi-Modal Media Manipulation Detection through Face-Enhanced Local Analysis

TL;DR

The paper tackles the challenge of detecting multi-modal media manipulation by foregrounding localized facial information alongside global image-text cues. It introduces M4-BLIP, a BLIP-2 based framework that extracts global and local features, aligns them with Fine-grained Contrastive Alignment, and fuses them via a Multi-modal Local-and-Global Fusion module using Q-Former, while also enabling interpretable outputs through integration with a large language model. Key contributions include the local-prior enhancement, cross-modal alignment and fusion architecture, and the end-to-end training scheme with dedicated detection heads and LLM-based explanations. Experimental results on the DGM^4 dataset show substantial performance gains over state-of-the-art methods and provide qualitative visualizations of attention and LLM reasoning, underscoring both improved accuracy and interpretability for practical forgery detection.

Abstract

In the contemporary digital landscape, multi-modal media manipulation has emerged as a significant societal threat, impacting the reliability and integrity of information dissemination. Current detection methodologies in this domain often overlook the crucial aspect of localized information, despite the fact that manipulations frequently occur in specific areas, particularly in facial regions. In response to this critical observation, we propose the M4-BLIP framework. This innovative framework utilizes the BLIP-2 model, renowned for its ability to extract local features, as the cornerstone for feature extraction. Complementing this, we incorporate local facial information as prior knowledge. A specially designed alignment and fusion module within M4-BLIP meticulously integrates these local and global features, creating a harmonious blend that enhances detection accuracy. Furthermore, our approach seamlessly integrates with Large Language Models (LLM), significantly improving the interpretability of the detection outcomes. Extensive quantitative and visualization experiments validate the effectiveness of our framework against the state-of-the-art competitors.

Figures (6)

• Figure 1: Examples of multi-modal manipulation detection (FS:Face Swap Manipulation, FA:aspect Attribute Manipulation, TS:Text Swap Manipulation, TA:Text Attribute Manipulation)
• Figure 2: Overview of proposed methods. Different levels of image features are extracted through a global image encoder and a local deepfake detector. Subsequently, Q-Former is employed to aggregate these features from various levels along with text features. Finally, tasks are executed in a layered manner using different detection heads, combined with LLM, to generate task-related descriptions. (Best viewed in color.)
• Figure 3: Visualization of attention map. The red markings indicate that this text carries more attention, while the blue boxes denote text that has been manipulated.
• Figure 4: Visualization of LLM output. The left side displays images, corresponding text, and multi-modal manipulation labels, the right side showcases the output of the LLM without finetuning and the LLM finetuning with DGM$^4$. The yellow boxes represent questions input by humans, while the blue boxes denote responses provided by the LLM.
• Figure 5: Implementation details of Q-Former.