Multi-modal Deepfake Detection and Localization with FPN-Transformer

TL;DR

The paper tackles the challenge of robust cross-modal deepfake detection and precise temporal localization by proposing a Feature Pyramid-Transformer (FPN-Transformer). It harnesses frozen pretrained encoders—WavLM for audio and CLIP for video—and builds a multi-scale temporal feature pyramid with R-TLM blocks and localized attention, enabling joint analysis of cross-modal temporal dependencies. A dual-branch predictor simultaneously outputs forgery probabilities and refines start/end offsets to achieve frame-level localization, trained with a focal classification loss and a DIoU-based regression loss. Evaluated on the IJCAI'25 DDL-AV benchmark, the method achieves a final score of 0.7535, demonstrating strong cross-modal generalization and precise localization, and providing a scalable approach for detecting sophisticated, fine-grained deepfakes.

Abstract

The rapid advancement of generative adversarial networks (GANs) and diffusion models has enabled the creation of highly realistic deepfake content, posing significant threats to digital trust across audio-visual domains. While unimodal detection methods have shown progress in identifying synthetic media, their inability to leverage cross-modal correlations and precisely localize forged segments limits their practicality against sophisticated, fine-grained manipulations. To address this, we introduce a multi-modal deepfake detection and localization framework based on a Feature Pyramid-Transformer (FPN-Transformer), addressing critical gaps in cross-modal generalization and temporal boundary regression. The proposed approach utilizes pre-trained self-supervised models (WavLM for audio, CLIP for video) to extract hierarchical temporal features. A multi-scale feature pyramid is constructed through R-TLM blocks with localized attention mechanisms, enabling joint analysis of cross-context temporal dependencies. The dual-branch prediction head simultaneously predicts forgery probabilities and refines temporal offsets of manipulated segments, achieving frame-level localization precision. We evaluate our approach on the test set of the IJCAI'25 DDL-AV benchmark, showing a good performance with a final score of 0.7535 for cross-modal deepfake detection and localization in challenging environments. Experimental results confirm the effectiveness of our approach and provide a novel way for generalized deepfake detection. Our code is available at https://github.com/Zig-HS/MM-DDL

Figures (2)

• Figure 1: Framework of proposed dual FPN-Transformer detection method. First, we employ pre-trained self-supervised models (WavLM/CLIP:ViT) to encode input temporal data into feature embeddings. Then, these embeddings are processed through an encoder composed of Transformer blocks to obtain the feature pyramid. And finally, prediction heads jointly analyze temporal features to predict forgery boundaries. We train two models separately for audios and videos, and combine the output results.
• Figure 2: Visualization results of our proposed method. Red represents forged segments, and yellow represents our predicted results. Our method can accurately predict the presence of forged video and audio segments in the samples for both video and audio modalities.