Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Audio-Visual Deepfake Detection With Local Temporal Inconsistencies

Marcella Astrid, Enjie Ghorbel, Djamila Aouada

TL;DR

The paper tackles audio-visual deepfake detection by targeting fine-grained temporal inconsistencies between modalities. It introduces a dual strategy: temporally-local pseudo-fake data augmentation and an architecture that computes per-time-step audio-visual distances, enhanced by cross-modal attention, to classify authenticity. Empirical results on DFDC and FakeAVCeleb demonstrate superior performance over state-of-the-art methods, highlighting the effectiveness of temporal-local cues over purely spatial cues. The approach offers practical value for robust detection in real-world scenarios where temporal misalignments are subtle yet informative.

Abstract

This paper proposes an audio-visual deepfake detection approach that aims to capture fine-grained temporal inconsistencies between audio and visual modalities. To achieve this, both architectural and data synthesis strategies are introduced. From an architectural perspective, a temporal distance map, coupled with an attention mechanism, is designed to capture these inconsistencies while minimizing the impact of irrelevant temporal subsequences. Moreover, we explore novel pseudo-fake generation techniques to synthesize local inconsistencies. Our approach is evaluated against state-of-the-art methods using the DFDC and FakeAVCeleb datasets, demonstrating its effectiveness in detecting audio-visual deepfakes.

Audio-Visual Deepfake Detection With Local Temporal Inconsistencies

TL;DR

The paper tackles audio-visual deepfake detection by targeting fine-grained temporal inconsistencies between modalities. It introduces a dual strategy: temporally-local pseudo-fake data augmentation and an architecture that computes per-time-step audio-visual distances, enhanced by cross-modal attention, to classify authenticity. Empirical results on DFDC and FakeAVCeleb demonstrate superior performance over state-of-the-art methods, highlighting the effectiveness of temporal-local cues over purely spatial cues. The approach offers practical value for robust detection in real-world scenarios where temporal misalignments are subtle yet informative.

Abstract

This paper proposes an audio-visual deepfake detection approach that aims to capture fine-grained temporal inconsistencies between audio and visual modalities. To achieve this, both architectural and data synthesis strategies are introduced. From an architectural perspective, a temporal distance map, coupled with an attention mechanism, is designed to capture these inconsistencies while minimizing the impact of irrelevant temporal subsequences. Moreover, we explore novel pseudo-fake generation techniques to synthesize local inconsistencies. Our approach is evaluated against state-of-the-art methods using the DFDC and FakeAVCeleb datasets, demonstrating its effectiveness in detecting audio-visual deepfakes.
Paper Structure (18 sections, 7 equations, 3 figures, 4 tables)

This paper contains 18 sections, 7 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Data augmentation
  4. Temporally-local manipulation
  5. Manipulation types
  6. Architectural design
  7. Feature extractors
  8. Temporal local distance
  9. Attention mechanism
  10. Classifier
  11. Experiments
  12. Experimental setup
  13. Ablation study
  14. Pseudo-fake manipulation
  15. Attention
  16. ...and 3 more sections

Figures (3)

  • Figure 1: We address temporal fine-grained inconsistencies from (a) the data synthesis perspective and (b) the architectural perspective. (a) We augment the original data with pseudo-fake examples by locally modifying the data in the temporal domain. (b) The classifier evaluates audio-visual inconsistencies at each temporal step.
  • Figure 2: Given (a) an original data sequence , we generate pseudo-fake data by locally modifying the sequence. In this example , we modify $\mathbf A_3$ to $\mathbf A_6$. We explore various modifications: (b) replacing with sub-sequences from another data sample , (c) repeating , (d) flipping , and (e-f) translating from the left or right.
  • Figure 3: Our architecture uses a fine-grained distance map for each time step of the extracted features , combined with an attention mechanism , to classify whether the input pair is fake.