Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mining Forgery Traces from Reconstruction Error: A Weakly Supervised Framework for Multimodal Deepfake Temporal Localization

Midou Guo, Qilin Yin, Wei Lu, Xiangyang Luo, Rui Yang

TL;DR

This work tackles fine-grained temporal localization of multimodal deepfakes under weak supervision by pivoting from modeling diverse forgery patterns to learning the intrinsic spatiotemporal regularities of authentic data. It introduces RT-DeepLoc, a framework that uses a Forgery Discovery Network based on a Masked Autoencoder trained solely on authentic videos to generate reconstruction-error cues for localization. To exploit these cues, it integrates Asymmetric Intra-video Contrastive Loss to focus on forgery hotspots and maintain robust authentic-feature compactness, along with Multi-task Learning Reinforcement to stabilize cross-stream predictions. Across large-scale datasets, RT-DeepLoc achieves state-of-the-art weakly supervised performance and demonstrates strong cross-dataset generalization, underscoring the practical value of reconstruction-based signals for robust deepfake localization.

Abstract

Modern deepfakes have evolved into localized and intermittent manipulations that require fine-grained temporal localization. The prohibitive cost of frame-level annotation makes weakly supervised methods a practical necessity, which rely only on video-level labels. To this end, we propose Reconstruction-based Temporal Deepfake Localization (RT-DeepLoc), a weakly supervised temporal forgery localization framework that identifies forgeries via reconstruction errors. Our framework uses a Masked Autoencoder (MAE) trained exclusively on authentic data to learn its intrinsic spatiotemporal patterns; this allows the model to produce significant reconstruction discrepancies for forged segments, effectively providing the missing fine-grained cues for localization. To robustly leverage these indicators, we introduce a novel Asymmetric Intra-video Contrastive Loss (AICL). By focusing on the compactness of authentic features guided by these reconstruction cues, AICL establishes a stable decision boundary that enhances local discrimination while preserving generalization to unseen forgeries. Extensive experiments on large-scale datasets, including LAV-DF, demonstrate that RT-DeepLoc achieves state-of-the-art performance in weakly-supervised temporal forgery localization.

Mining Forgery Traces from Reconstruction Error: A Weakly Supervised Framework for Multimodal Deepfake Temporal Localization

TL;DR

This work tackles fine-grained temporal localization of multimodal deepfakes under weak supervision by pivoting from modeling diverse forgery patterns to learning the intrinsic spatiotemporal regularities of authentic data. It introduces RT-DeepLoc, a framework that uses a Forgery Discovery Network based on a Masked Autoencoder trained solely on authentic videos to generate reconstruction-error cues for localization. To exploit these cues, it integrates Asymmetric Intra-video Contrastive Loss to focus on forgery hotspots and maintain robust authentic-feature compactness, along with Multi-task Learning Reinforcement to stabilize cross-stream predictions. Across large-scale datasets, RT-DeepLoc achieves state-of-the-art weakly supervised performance and demonstrates strong cross-dataset generalization, underscoring the practical value of reconstruction-based signals for robust deepfake localization.

Abstract

Modern deepfakes have evolved into localized and intermittent manipulations that require fine-grained temporal localization. The prohibitive cost of frame-level annotation makes weakly supervised methods a practical necessity, which rely only on video-level labels. To this end, we propose Reconstruction-based Temporal Deepfake Localization (RT-DeepLoc), a weakly supervised temporal forgery localization framework that identifies forgeries via reconstruction errors. Our framework uses a Masked Autoencoder (MAE) trained exclusively on authentic data to learn its intrinsic spatiotemporal patterns; this allows the model to produce significant reconstruction discrepancies for forged segments, effectively providing the missing fine-grained cues for localization. To robustly leverage these indicators, we introduce a novel Asymmetric Intra-video Contrastive Loss (AICL). By focusing on the compactness of authentic features guided by these reconstruction cues, AICL establishes a stable decision boundary that enhances local discrimination while preserving generalization to unseen forgeries. Extensive experiments on large-scale datasets, including LAV-DF, demonstrate that RT-DeepLoc achieves state-of-the-art performance in weakly-supervised temporal forgery localization.
Paper Structure (19 sections, 4 equations, 4 figures, 3 tables)

This paper contains 19 sections, 4 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Weakly supervised Learning
  4. Mask Autoencoder
  5. Proposed Method
  6. Overview
  7. Forgery Discovery Network based on MAE
  8. Asymmetric Intra-video Contrastive Loss
  9. Multi-task Learning Reinforcement
  10. Experiments
  11. Experimental Settings
  12. Implementation details
  13. Intra-Dataset Evaluation
  14. Cross-Dataset Evaluation
  15. Ablation
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Comparison of different temporal forgery localization tasks: (a) Fully supervised temporal forgery localization; (b) Mutimodal weakly supervised temporal forgery localization.
  • Figure 2: (a) The overall workflow and data flow of the proposed framework. (b) The internal architecture of the core components within RT-DeepLoc, which includes the Multimodal Feature Encoding and Fusion module, the Forgery Discovery Network based on MAE, the Asymmetric Intra-video Contrastive Loss module, and the Multi-task Learning Reinforcement strategy.
  • Figure 3: Sensitivity analysis of hyperparameters on the LAV-DF dataset. (a) The effect of the number of selected frames $K$ in the AICL module. (b) The effect of the masking ratio $\rho$ in the FDN module.
  • Figure 4: Qualitative visualization of modality-specific reconstruction discrepancies on LAV-DF. We present four scenarios: (a) audio-only, (b) multimodal, (c) visual-only forgeries, and (d) authentic video. Blue and green curves represent visual and audio reconstruction errors, respectively, while shaded areas indicate ground-truth intervals.