Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AuViRe: Audio-visual Speech Representation Reconstruction for Deepfake Temporal Localization

Christos Koutlis, Symeon Papadopoulos

TL;DR

This work presents a novel approach to temporal localization of deepfakes by leveraging Audio-Visual Speech Representation Reconstruction (AuViRe), which reconstructs speech representations from one modality based on the other based on the other, thereby providing robust discriminative cues for precise temporal forgery localization.

Abstract

With the rapid advancement of sophisticated synthetic audio-visual content, e.g., for subtle malicious manipulations, ensuring the integrity of digital media has become paramount. This work presents a novel approach to temporal localization of deepfakes by leveraging Audio-Visual Speech Representation Reconstruction (AuViRe). Specifically, our approach reconstructs speech representations from one modality (e.g., lip movements) based on the other (e.g., audio waveform). Cross-modal reconstruction is significantly more challenging in manipulated video segments, leading to amplified discrepancies, thereby providing robust discriminative cues for precise temporal forgery localization. AuViRe outperforms the state of the art by +8.9 AP@0.95 on LAV-DF, +9.6 AP@0.5 on AV-Deepfake1M, and +5.1 AUC on an in-the-wild experiment. Code available at https://github.com/mever-team/auvire.

AuViRe: Audio-visual Speech Representation Reconstruction for Deepfake Temporal Localization

TL;DR

This work presents a novel approach to temporal localization of deepfakes by leveraging Audio-Visual Speech Representation Reconstruction (AuViRe), which reconstructs speech representations from one modality based on the other based on the other, thereby providing robust discriminative cues for precise temporal forgery localization.

Abstract

With the rapid advancement of sophisticated synthetic audio-visual content, e.g., for subtle malicious manipulations, ensuring the integrity of digital media has become paramount. This work presents a novel approach to temporal localization of deepfakes by leveraging Audio-Visual Speech Representation Reconstruction (AuViRe). Specifically, our approach reconstructs speech representations from one modality (e.g., lip movements) based on the other (e.g., audio waveform). Cross-modal reconstruction is significantly more challenging in manipulated video segments, leading to amplified discrepancies, thereby providing robust discriminative cues for precise temporal forgery localization. AuViRe outperforms the state of the art by +8.9 AP@0.95 on LAV-DF, +9.6 AP@0.5 on AV-Deepfake1M, and +5.1 AUC on an in-the-wild experiment. Code available at https://github.com/mever-team/auvire.

Paper Structure

This paper contains 29 sections, 9 equations, 7 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Problem Formulation
  5. Visual and Audio Speech Representations
  6. Representation Reconstruction Module
  7. Reconstruction-Discrepancy Encoder
  8. Predictions
  9. Objective Function
  10. Experimental Setup
  11. Datasets
  12. Implementation Details
  13. Real-world Application Implementation
  14. Evaluation
  15. Results
  16. ...and 14 more sections

Figures (7)

  • Figure 1: The proposed AuViRe architecture.
  • Figure 2: Examples of correctly (left; https://www.youtube.com/shorts/xAPIjhkXF-0) and erroneously (right; https://www.youtube.com/shorts/D9mQGzG9dRk) classified real samples.
  • Figure 3: Robustness analysis using visual (cyan plots) and audio (yellow plots) distortions. Overall average robustness is also reported.
  • Figure 4: Hyperparameter grid results on LAV-DF. Best rank is highlighted with red.
  • Figure 5: Hyperparameter grid results on AV-Deepfake1M. Best rank is highlighted with red.
  • ...and 2 more figures