Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detecting Deepfakes with Multivariate Soft Blending and CLIP-based Image-Text Alignment

Jingwei Li, Jiaxin Tong, Pengfei Wu

TL;DR

A novel Multivariate and Soft Blending Augmentation with CLIP-guided Forgery Intensity Estimation (MSBA-CLIP) framework that synthesizes images by blending forgeries from multiple methods with random weights, forcing the model to learn generalizable patterns.

Abstract

The proliferation of highly realistic facial forgeries necessitates robust detection methods. However, existing approaches often suffer from limited accuracy and poor generalization due to significant distribution shifts among samples generated by diverse forgery techniques. To address these challenges, we propose a novel Multivariate and Soft Blending Augmentation with CLIP-guided Forgery Intensity Estimation (MSBA-CLIP) framework. Our method leverages the multimodal alignment capabilities of CLIP to capture subtle forgery traces. We introduce a Multivariate and Soft Blending Augmentation (MSBA) strategy that synthesizes images by blending forgeries from multiple methods with random weights, forcing the model to learn generalizable patterns. Furthermore, a dedicated Multivariate Forgery Intensity Estimation (MFIE) module is designed to explicitly guide the model in learning features related to varied forgery modes and intensities. Extensive experiments demonstrate state-of-the-art performance. On in-domain tests, our method improves Accuracy and AUC by 3.32\% and 4.02\%, respectively, over the best baseline. In cross-domain evaluations across five datasets, it achieves an average AUC gain of 3.27\%. Ablation studies confirm the efficacy of both proposed components. While the reliance on a large vision-language model entails higher computational cost, our work presents a significant step towards more generalizable and robust deepfake detection.

Detecting Deepfakes with Multivariate Soft Blending and CLIP-based Image-Text Alignment

TL;DR

A novel Multivariate and Soft Blending Augmentation with CLIP-guided Forgery Intensity Estimation (MSBA-CLIP) framework that synthesizes images by blending forgeries from multiple methods with random weights, forcing the model to learn generalizable patterns.

Abstract

The proliferation of highly realistic facial forgeries necessitates robust detection methods. However, existing approaches often suffer from limited accuracy and poor generalization due to significant distribution shifts among samples generated by diverse forgery techniques. To address these challenges, we propose a novel Multivariate and Soft Blending Augmentation with CLIP-guided Forgery Intensity Estimation (MSBA-CLIP) framework. Our method leverages the multimodal alignment capabilities of CLIP to capture subtle forgery traces. We introduce a Multivariate and Soft Blending Augmentation (MSBA) strategy that synthesizes images by blending forgeries from multiple methods with random weights, forcing the model to learn generalizable patterns. Furthermore, a dedicated Multivariate Forgery Intensity Estimation (MFIE) module is designed to explicitly guide the model in learning features related to varied forgery modes and intensities. Extensive experiments demonstrate state-of-the-art performance. On in-domain tests, our method improves Accuracy and AUC by 3.32\% and 4.02\%, respectively, over the best baseline. In cross-domain evaluations across five datasets, it achieves an average AUC gain of 3.27\%. Ablation studies confirm the efficacy of both proposed components. While the reliance on a large vision-language model entails higher computational cost, our work presents a significant step towards more generalizable and robust deepfake detection.
Paper Structure (39 sections, 6 equations, 5 figures, 7 tables)

This paper contains 39 sections, 6 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Traditional Blind Image Separation Methods
  4. Deep Learning-Based Approaches
  5. CNN-Based Architectures
  6. GAN-Based Frameworks
  7. Advanced Deep Learning Variants
  8. Diffusion Models in Image Processing
  9. Foundational Diffusion Architectures
  10. Applications in Image Restoration
  11. Specialized Diffusion Variants
  12. Foundational Models and OCR Toolkits.
  13. End-to-End Text Detection and Recognition.
  14. Document Structure Parsing and Layout Analysis.
  15. Text-Centric Multimodal Understanding and Reasoning.
  16. ...and 24 more sections

Figures (5)

  • Figure 1: Schematic diagram of the proposed MSBA-CLIP framework. The input image undergoes MSBA augmentation. The CLIP-based encoder fuses image and text prompts. The Multivariate Forgery Intensity Estimation (MFIE) module predicts both the spatial intensity map and the method-specific blending weights. The final prediction is a fusion of a classification head output and a text-image similarity score.
  • Figure 2: Detailed architecture of the Multivariate Forgery Intensity Estimation (MFIE) module. It takes patch-level features from the visual encoder, upsamples them, and estimates both a per-pixel forgery intensity map and the method-wise blending weights.
  • Figure 3: Illustration of the Multivariate and Soft Blending Augmentation (MSBA) process. (a) Step 1: Calculate per-method forgery intensity maps ($M_i$) between the real image and each forged version. (b) Step 2: Synthesize a blended intensity map $\tilde{M}$ by linearly combining individual maps with random weights $\alpha_i$. (c) Step 3: Generate the final blended forgery image $\tilde{I}$ by subtracting the scaled blended map from the real image.
  • Figure 4: Robustness evaluation under five types of image perturbations at varying intensity levels. Our method (MSBA-CLIP) demonstrates superior stability compared to state-of-the-art baselines.
  • Figure 5: Visualization of forgery intensity maps. (a) Input images (Real, Blended, DF, FF, FS, NT). (b) Ground-truth intensity maps (patch-level, multiplied by 5 for visibility). (c) Predicted intensity maps from our MFIE module. The model accurately localizes manipulated regions across various forgery types and complex blends.