Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PIA: Deepfake Detection Using Phoneme-Temporal and Identity-Dynamic Analysis

Soumyya Kanti Datta, Tanvi Ranga, Chengzhe Sun, Siwei Lyu

TL;DR

PIA tackles deepfake detection by integrating multiple cues across modalities: phoneme articulation, lip geometry, and identity dynamics. The method aligns phonemes to frames, extracts viseme crops, computes lip geometry, and uses ArcFace embeddings, all fused with cross-modal attention and an ArcFace temporal consistency loss to detect subtle audiovisual inconsistencies. It achieves state-of-the-art performance on FakeAVCeleb and DeepSpeak v2.0, with strong cross-manipulation generalization and ablation analyses highlighting the contribution of each modality. The work advances practical deepfake defense by leveraging temporal and cross-modal signals, with public code to facilitate adoption and further research.

Abstract

The rise of manipulated media has made deepfakes a particularly insidious threat, involving various generative manipulations such as lip-sync modifications, face-swaps, and avatar-driven facial synthesis. Conventional detection methods, which predominantly depend on manually designed phoneme-viseme alignment thresholds, fundamental frame-level consistency checks, or a unimodal detection strategy, inadequately identify modern-day deepfakes generated by advanced generative models such as GANs, diffusion models, and neural rendering techniques. These advanced techniques generate nearly perfect individual frames yet inadvertently create minor temporal discrepancies frequently overlooked by traditional detectors. We present a novel multimodal audio-visual framework, Phoneme-Temporal and Identity-Dynamic Analysis(PIA), incorporating language, dynamic face motion, and facial identification cues to address these limitations. We utilize phoneme sequences, lip geometry data, and advanced facial identity embeddings. This integrated method significantly improves the detection of subtle deepfake alterations by identifying inconsistencies across multiple complementary modalities. Code is available at https://github.com/skrantidatta/PIA

PIA: Deepfake Detection Using Phoneme-Temporal and Identity-Dynamic Analysis

TL;DR

PIA tackles deepfake detection by integrating multiple cues across modalities: phoneme articulation, lip geometry, and identity dynamics. The method aligns phonemes to frames, extracts viseme crops, computes lip geometry, and uses ArcFace embeddings, all fused with cross-modal attention and an ArcFace temporal consistency loss to detect subtle audiovisual inconsistencies. It achieves state-of-the-art performance on FakeAVCeleb and DeepSpeak v2.0, with strong cross-manipulation generalization and ablation analyses highlighting the contribution of each modality. The work advances practical deepfake defense by leveraging temporal and cross-modal signals, with public code to facilitate adoption and further research.

Abstract

The rise of manipulated media has made deepfakes a particularly insidious threat, involving various generative manipulations such as lip-sync modifications, face-swaps, and avatar-driven facial synthesis. Conventional detection methods, which predominantly depend on manually designed phoneme-viseme alignment thresholds, fundamental frame-level consistency checks, or a unimodal detection strategy, inadequately identify modern-day deepfakes generated by advanced generative models such as GANs, diffusion models, and neural rendering techniques. These advanced techniques generate nearly perfect individual frames yet inadvertently create minor temporal discrepancies frequently overlooked by traditional detectors. We present a novel multimodal audio-visual framework, Phoneme-Temporal and Identity-Dynamic Analysis(PIA), incorporating language, dynamic face motion, and facial identification cues to address these limitations. We utilize phoneme sequences, lip geometry data, and advanced facial identity embeddings. This integrated method significantly improves the detection of subtle deepfake alterations by identifying inconsistencies across multiple complementary modalities. Code is available at https://github.com/skrantidatta/PIA
Paper Structure (12 sections, 6 equations, 4 figures, 3 tables)

This paper contains 12 sections, 6 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Backgrounds
  4. Method
  5. Feature Extraction
  6. Model Architecture
  7. ArcFace Temporal Consistency Loss
  8. Experiments
  9. Experimental settings
  10. Results
  11. Ablation Analysis
  12. Conclusion

Figures (4)

  • Figure 1: Comparison of lip shapes corresponding to different phonemes. Note that in lip-sync deepfake videos, the degree of lip closure often does not correspond to the sound being pronounced.
  • Figure 2: Temporal drift in ArcFace embeddings between consecutive frames for a real video (green) and its face-swap deepfake counterpart (red). Higher drift in initial frames and consistently higher overall drift in fake samples indicates temporal identity inconsistencies.
  • Figure 3: End-to-end pipeline of our proposed PIA model. It consists of (1) Multimodal Feature Extraction and Temporal Alignment, and (2) Cross-Modal Temporal Deepfake Detector.
  • Figure 4: Example frames from the FakeAVCeleb dataset (left) and the DeepSpeak v2.0 dataset (right). The top row shows frames from lip-sync deepfakes, while the bottom row presents frames from face-swap deepfakes. Notable differences in native resolution and visual quality are apparent. For comparison, face regions from FakeAVCeleb were enlarged to match the scale of DeepSpeak v2.0 samples.