Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CapsFake: A Multimodal Capsule Network for Detecting Instruction-Guided Deepfakes

Tuan Nguyen, Naseem Khan, Issa Khalil

TL;DR

CapsFake tackles instruction-guided deepfakes by presenting a multimodal capsule network that fuses visual, textual, and frequency information through dynamic routing. By encoding semantic consistency across modalities into high-level capsules, it localizes manipulated regions while maintaining strong global accuracy. Across MagicBrush, Unsplash Edits, Open Images Edits, and Multi-turn Edits, CapsFake outperforms state-of-the-art detectors and shows robust resilience to natural distortions, white-box and black-box adversaries, and unseen domain shifts. The approach offers interpretable saliency and routing visualizations, making it a practical and trustworthy tool for forensic integrity in digital media pipelines.

Abstract

The rapid evolution of deepfake technology, particularly in instruction-guided image editing, threatens the integrity of digital images by enabling subtle, context-aware manipulations. Generated conditionally from real images and textual prompts, these edits are often imperceptible to both humans and existing detection systems, revealing significant limitations in current defenses. We propose a novel multimodal capsule network, CapsFake, designed to detect such deepfake image edits by integrating low-level capsules from visual, textual, and frequency-domain modalities. High-level capsules, predicted through a competitive routing mechanism, dynamically aggregate local features to identify manipulated regions with precision. Evaluated on diverse datasets, including MagicBrush, Unsplash Edits, Open Images Edits, and Multi-turn Edits, CapsFake outperforms state-of-the-art methods by up to 20% in detection accuracy. Ablation studies validate its robustness, achieving detection rates above 94% under natural perturbations and 96% against adversarial attacks, with excellent generalization to unseen editing scenarios. This approach establishes a powerful framework for countering sophisticated image manipulations.

CapsFake: A Multimodal Capsule Network for Detecting Instruction-Guided Deepfakes

TL;DR

CapsFake tackles instruction-guided deepfakes by presenting a multimodal capsule network that fuses visual, textual, and frequency information through dynamic routing. By encoding semantic consistency across modalities into high-level capsules, it localizes manipulated regions while maintaining strong global accuracy. Across MagicBrush, Unsplash Edits, Open Images Edits, and Multi-turn Edits, CapsFake outperforms state-of-the-art detectors and shows robust resilience to natural distortions, white-box and black-box adversaries, and unseen domain shifts. The approach offers interpretable saliency and routing visualizations, making it a practical and trustworthy tool for forensic integrity in digital media pipelines.

Abstract

The rapid evolution of deepfake technology, particularly in instruction-guided image editing, threatens the integrity of digital images by enabling subtle, context-aware manipulations. Generated conditionally from real images and textual prompts, these edits are often imperceptible to both humans and existing detection systems, revealing significant limitations in current defenses. We propose a novel multimodal capsule network, CapsFake, designed to detect such deepfake image edits by integrating low-level capsules from visual, textual, and frequency-domain modalities. High-level capsules, predicted through a competitive routing mechanism, dynamically aggregate local features to identify manipulated regions with precision. Evaluated on diverse datasets, including MagicBrush, Unsplash Edits, Open Images Edits, and Multi-turn Edits, CapsFake outperforms state-of-the-art methods by up to 20% in detection accuracy. Ablation studies validate its robustness, achieving detection rates above 94% under natural perturbations and 96% against adversarial attacks, with excellent generalization to unseen editing scenarios. This approach establishes a powerful framework for countering sophisticated image manipulations.
Paper Structure (42 sections, 12 equations, 9 figures, 18 tables, 1 algorithm)

This paper contains 42 sections, 12 equations, 9 figures, 18 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Threat Model and Security Objectives
  3. Achieving High Detection Performance on Semantic Deepfake Manipulations.
  4. Robustness to Natural Distortions and Adversarial Perturbations.
  5. Generalization to Unseen Domains.
  6. Localization of Semantic Manipulations.
  7. Related Work
  8. Unimodal Detection Approaches
  9. Multi-modal Detection Frameworks
  10. Adversarial Robustness Analysis
  11. Security Limitations in Spatial Understanding
  12. Human Perception and Security Implications
  13. Background
  14. Our Approach: CapsFake
  15. Overview
  16. ...and 27 more sections

Figures (9)

  • Figure 1: Real (first row) and fake (second row) image editing pairs, sourced from the Unsplash Edits (a), the Open Images Edits (b), MagicBrush (c), and MagicBrush (d).
  • Figure 2: Malicious Image Manipulation Pipeline. A threat actor uses generative AI tools to manipulate specific elements of an image, leveraging image translation and understanding models to guide semantic edits. These capabilities facilitate identity obfuscation, impersonation, and disinformation.
  • Figure 3: Overview of the proposed multimodal capsule network, CapsFake, for detecting instruction-guided deepfakes.
  • Figure 4: Examples of various natural perturbations analyzed in our study.
  • Figure 5: Detection performance (F1 score) under progressive multi-turn semantic edits.
  • ...and 4 more figures