Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CAST: Cross-Attentive Spatio-Temporal feature fusion for deepfake detection

Aryan Thakre, Omkar Nagwekar, Vedang Talekar, Aparna Santra Biswas

Abstract

Deepfakes have emerged as a significant threat to digital media authenticity, increasing the need for advanced detection techniques that can identify subtle and time-dependent manipulations. CNNs are effective at capturing spatial artifacts and Transformers excel at modeling temporal inconsistencies. However, many existing CNN-Transformer models process spatial and temporal features independently. In particular, attention based methods often use independent attention mechanisms for spatial and temporal features and combine them using naive approaches like averaging, addition or concatenation, limiting the depth of spatio-temporal interaction. To address this challenge, we propose a unified CAST model that leverages cross-attention to effectively fuse spatial and temporal features in a more integrated manner. Our approach allows temporal features to dynamically attend to relevant spatial regions, enhancing the model's ability to detect fine-grained, time-evolving artifacts such as flickering eyes or warped lips. This design enables more precise localization and deeper contextual understanding, leading to improved performance across diverse and challenging scenarios. We evaluate the performance of our model using the FaceForensics++, Celeb-DF, DeepfakeDetection and Deepfake Detection Challenge (DFDC) datasets in both intra and cross dataset settings to affirm the superiority of our approach. Our model achieves strong performance with an Area Under the Curve (AUC) of 99.49% and an accuracy of 97.57% in intra-dataset evaluations. In cross-dataset testing, the model achieves AUC scores of 93.31% and 81.25% on the unseen DeepFakeDetection and DFDC datasets, respectively. These results highlight the effectiveness of cross-attention-based feature fusion in enhancing the robustness of deepfake video detection.

CAST: Cross-Attentive Spatio-Temporal feature fusion for deepfake detection

Abstract

Deepfakes have emerged as a significant threat to digital media authenticity, increasing the need for advanced detection techniques that can identify subtle and time-dependent manipulations. CNNs are effective at capturing spatial artifacts and Transformers excel at modeling temporal inconsistencies. However, many existing CNN-Transformer models process spatial and temporal features independently. In particular, attention based methods often use independent attention mechanisms for spatial and temporal features and combine them using naive approaches like averaging, addition or concatenation, limiting the depth of spatio-temporal interaction. To address this challenge, we propose a unified CAST model that leverages cross-attention to effectively fuse spatial and temporal features in a more integrated manner. Our approach allows temporal features to dynamically attend to relevant spatial regions, enhancing the model's ability to detect fine-grained, time-evolving artifacts such as flickering eyes or warped lips. This design enables more precise localization and deeper contextual understanding, leading to improved performance across diverse and challenging scenarios. We evaluate the performance of our model using the FaceForensics++, Celeb-DF, DeepfakeDetection and Deepfake Detection Challenge (DFDC) datasets in both intra and cross dataset settings to affirm the superiority of our approach. Our model achieves strong performance with an Area Under the Curve (AUC) of 99.49% and an accuracy of 97.57% in intra-dataset evaluations. In cross-dataset testing, the model achieves AUC scores of 93.31% and 81.25% on the unseen DeepFakeDetection and DFDC datasets, respectively. These results highlight the effectiveness of cross-attention-based feature fusion in enhancing the robustness of deepfake video detection.

Paper Structure

This paper contains 29 sections, 23 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Deepfake Generation
  4. Deepfake Detection
  5. Method
  6. Frame Selection and Preprocessing
  7. Spatial Resizing and Normalization
  8. Feature Extraction via CNN Backbone
  9. Temporal Token Computation and Encoding
  10. Feature Fusion via Cross-Attention
  11. Classification Layer and Optimization
  12. Experiments
  13. Datasets
  14. Experimental settings
  15. Evaluation Metrics:
  16. ...and 14 more sections

Figures (6)

  • Figure 1: The pipeline of our proposed framework CAST which consists of an EfficientNet-Transformer hybrid architecture with cross-attention based spatio-temporal fusion.
  • Figure 2: The working of Transformer encoder (single head) modeling temporal dependencies using features extracted from EfficentNet layers
  • Figure 3: Working of Cross-Attention module in the CAST Model.
  • Figure 4: Qualitative analysis on FF++(HQ) manipulation types: Deepfakes (DF), Face2Face (F2F), FaceSwap (FS), and NeuralTextures (NT). From left to right is the real frame, the corresponding fake frame, and the attention heatmap generated by three model variants: without cross-attention, with the proposed cross-attention mechanism, and with decoupled attention fusion.
  • Figure 5: Visual comparison of CNN feature map activations with and without the $1 \times 1$ projection layer across different datasets. Each row corresponds to a sample from a manipulated video (FF++, DFD, and Celeb-DF v2).
  • ...and 1 more figures