Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Novel Unified Approach to Deepfake Detection

Lord Sen, Shyamapada Mukherjee

TL;DR

The paper tackles the challenge of reliable Deepfake detection in images and videos by proposing a unified, multi-modal architecture that fuses spatial and frequency-domain cues through cross-stream attention. It introduces a dual-stream feature encoder (spatial CNN/transformer and frequency transformer), a blood-detection module, a Cross-Stream Attention Fusion (CSAF) mechanism, and a Class Token Refinement Module with multiscale patch embedding to produce a robust final decision. The approach achieves state-of-the-art results on major benchmarks (e.g., 99.80% AUC on FF++ and 99.88% on Celeb-DF with Swin+BERT) and demonstrates strong cross-dataset generalization, signaling practical impact for multimedia verification. The combination of spatial-frequency fusion, biometric-inspired cues, and multi-scale tokenization provides a resilient framework against diverse deepfake generation techniques, with potential applications in digital forensics and content authentication.

Abstract

The advancements in the field of AI is increasingly giving rise to various threats. One of the most prominent of them is the synthesis and misuse of Deepfakes. To sustain trust in this digital age, detection and tagging of deepfakes is very necessary. In this paper, a novel architecture for Deepfake detection in images and videos is presented. The architecture uses cross attention between spatial and frequency domain features along with a blood detection module to classify an image as real or fake. This paper aims to develop a unified architecture and provide insights into each step. Though this approach we achieve results better than SOTA, specifically 99.80%, 99.88% AUC on FF++ and Celeb-DF upon using Swin Transformer and BERT and 99.55, 99.38 while using EfficientNet-B4 and BERT. The approach also generalizes very well achieving great cross dataset results as well.

A Novel Unified Approach to Deepfake Detection

TL;DR

The paper tackles the challenge of reliable Deepfake detection in images and videos by proposing a unified, multi-modal architecture that fuses spatial and frequency-domain cues through cross-stream attention. It introduces a dual-stream feature encoder (spatial CNN/transformer and frequency transformer), a blood-detection module, a Cross-Stream Attention Fusion (CSAF) mechanism, and a Class Token Refinement Module with multiscale patch embedding to produce a robust final decision. The approach achieves state-of-the-art results on major benchmarks (e.g., 99.80% AUC on FF++ and 99.88% on Celeb-DF with Swin+BERT) and demonstrates strong cross-dataset generalization, signaling practical impact for multimedia verification. The combination of spatial-frequency fusion, biometric-inspired cues, and multi-scale tokenization provides a resilient framework against diverse deepfake generation techniques, with potential applications in digital forensics and content authentication.

Abstract

The advancements in the field of AI is increasingly giving rise to various threats. One of the most prominent of them is the synthesis and misuse of Deepfakes. To sustain trust in this digital age, detection and tagging of deepfakes is very necessary. In this paper, a novel architecture for Deepfake detection in images and videos is presented. The architecture uses cross attention between spatial and frequency domain features along with a blood detection module to classify an image as real or fake. This paper aims to develop a unified architecture and provide insights into each step. Though this approach we achieve results better than SOTA, specifically 99.80%, 99.88% AUC on FF++ and Celeb-DF upon using Swin Transformer and BERT and 99.55, 99.38 while using EfficientNet-B4 and BERT. The approach also generalizes very well achieving great cross dataset results as well.
Paper Structure (15 sections, 28 equations, 3 figures, 2 tables)

This paper contains 15 sections, 28 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Architecture
  4. Input Layer
  5. Preprocessing Layer
  6. Dual-Stream Feature Encoder
  7. Spatial Feature Encoder
  8. Frequency Feature Encoder
  9. Detection of Blood underneath the skin
  10. Cross-Stream Attention Fusion (CSAF)
  11. Multiscale Patch Embedding
  12. Class Token Refinement Module (CTRM)
  13. Final Classification Head
  14. Results
  15. Conclusion

Figures (3)

  • Figure 1: The graphs of Energy, entropy and PSD of three different sets of real(Blue) and fake(Red) images.
  • Figure 2: Our Proposed Architecture-1 and 2
  • Figure 3: The figure shows the outputs of various layers, when an image is passed to our proposed architecture.