Can Current Detectors Catch Face-to-Voice Deepfake Attacks?

TL;DR

FOICE enables voice synthesis from a single facial image, bypassing voice samples and threatening biometric authentication. The authors evaluate four detectors across FOICE and SpeechT5, under clean, noisy, and denoised conditions, and propose a tailored fine-tuning approach. They find that detectors fail to detect FOICE out-of-the-box, but FOICE-focused fine-tuning dramatically improves in-distribution accuracy, sometimes to near 0 $EER$, while risking poorer generalization to unseen generators. The study emphasizes the need for domain-robust, cross-modal representations and larger diverse datasets for durable defenses.

Abstract

The rapid advancement of generative models has enabled the creation of increasingly stealthy synthetic voices, commonly referred to as audio deepfakes. A recent technique, FOICE [USENIX'24], demonstrates a particularly alarming capability: generating a victim's voice from a single facial image, without requiring any voice sample. By exploiting correlations between facial and vocal features, FOICE produces synthetic voices realistic enough to bypass industry-standard authentication systems, including WeChat Voiceprint and Microsoft Azure. This raises serious security concerns, as facial images are far easier for adversaries to obtain than voice samples, dramatically lowering the barrier to large-scale attacks. In this work, we investigate two core research questions: (RQ1) can state-of-the-art audio deepfake detectors reliably detect FOICE-generated speech under clean and noisy conditions, and (RQ2) whether fine-tuning these detectors on FOICE data improves detection without overfitting, thereby preserving robustness to unseen voice generators such as SpeechT5. Our study makes three contributions. First, we present the first systematic evaluation of FOICE detection, showing that leading detectors consistently fail under both standard and noisy conditions. Second, we introduce targeted fine-tuning strategies that capture FOICE-specific artifacts, yielding significant accuracy improvements. Third, we assess generalization after fine-tuning, revealing trade-offs between specialization to FOICE and robustness to unseen synthesis pipelines. These findings expose fundamental weaknesses in today's defenses and motivate new architectures and training protocols for next-generation audio deepfake detection.

Figures (6)

• Figure 1: Illustration of Face to Voice Deepfake
• Figure 2: Equal Error Rate (EER) of the four detectors—AASIST, Ren et al., Sun et al., and TCM—on FOICE data under three audio conditions (original, noisy, denoised). Bars show baseline (solid) and FOICE-fine-tuned (hatched) models. Lower EER indicates stronger discrimination between real and fake audio. Fine-tuning yields large reductions in EER across all conditions, with AASIST and Ren et al. approaching near-zero values. TCM shows the least improvement but still benefits.
• Figure 3: Mel spectrograms comparing audio from three pipelines: (left) AVSpeech (real), (middle) FOICE generated with the SV2TTS vocoder, and (right) SpeechT5 generated with HiFi-GAN vocoder. Each spectrogram shows distinct time–frequency signatures and synthesis artifacts. FOICE, in particular, exhibits banding and smoothing effects absent from AVSpeech’s richer harmonic structures, illustrating how cross-vocoder differences create challenges for robust deepfake detection.
• Figure 4: Accuracy of four detectors under baseline (solid) and FOICE‑fine‑tuned (hatched) regimes across two pipelines: FOICE with SV2TTS (blue) and SpeechT5 with HiFi‑GAN (red). The results highlight a trade‑off between in‑distribution adaptation and generalization to unseen synthesis methods when optimizing for accuracy.
• Figure 5: Accuracy of the four detectors on FOICE originals, noisy, and denoised audio. Solid bars show baseline performance, and hatched bars show fine-tuned variants. After fine-tuning, AASIST and Ren et al. AISVGeneralization achieve near-perfect accuracy across all conditions, while Sun et al. VocoderArtifacts and TCM improve substantially, though remain weaker overall. The results highlight the effectiveness of fine-tuning in boosting in-distribution detection, even under degraded audio conditions.