Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VocalBridge: Latent Diffusion-Bridge Purification for Defeating Perturbation-Based Voiceprint Defenses

Maryam Abbasihafshejani, AHM Nazmus Sakib, Murtuza Jadliwala

TL;DR

VocalBridge reveals the fragility of perturbation-based voice protections by demonstrating a latent-diffusion bridge purifier that recovers clean speaker characteristics from defended speech within the EnCodec latent space. By integrating a time-conditioned 1D U‑Net and a Whisper-guided phoneme conditioning, the method achieves transcript-free purification with strong restoration of verifiability while preserving perceptual quality. Across diverse defenses, synthesis tools, and ASV back-ends, VocalBridge and its Whisper-enhanced variant consistently outperform baselines, and remain robust under cross-perturbation and adaptive attacks, underscoring the need for defenses that anticipate purification. The work contributes a thorough evaluation framework, a practical purification approach, and a call for more resilient protections against evolving voice-cloning threats with broad security and privacy implications.

Abstract

The rapid advancement of speech synthesis technologies, including text-to-speech (TTS) and voice conversion (VC), has intensified security and privacy concerns related to voice cloning. Recent defenses attempt to prevent unauthorized cloning by embedding protective perturbations into speech to obscure speaker identity while maintaining intelligibility. However, adversaries can apply advanced purification techniques to remove these perturbations, recover authentic acoustic characteristics, and regenerate cloneable voices. Despite the growing realism of such attacks, the robustness of existing defenses under adaptive purification remains insufficiently studied. Most existing purification methods are designed to counter adversarial noise in automatic speech recognition (ASR) systems rather than speaker verification or voice cloning pipelines. As a result, they fail to suppress the fine-grained acoustic cues that define speaker identity and are often ineffective against speaker verification attacks (SVA). To address these limitations, we propose Diffusion-Bridge (VocalBridge), a purification framework that learns a latent mapping from perturbed to clean speech in the EnCodec latent space. Using a time-conditioned 1D U-Net with a cosine noise schedule, the model enables efficient, transcript-free purification while preserving speaker-discriminative structure. We further introduce a Whisper-guided phoneme variant that incorporates lightweight temporal guidance without requiring ground-truth transcripts. Experimental results show that our approach consistently outperforms existing purification methods in recovering cloneable voices from protected speech. Our findings demonstrate the fragility of current perturbation-based defenses and highlight the need for more robust protection mechanisms against evolving voice-cloning and speaker verification threats.

VocalBridge: Latent Diffusion-Bridge Purification for Defeating Perturbation-Based Voiceprint Defenses

TL;DR

VocalBridge reveals the fragility of perturbation-based voice protections by demonstrating a latent-diffusion bridge purifier that recovers clean speaker characteristics from defended speech within the EnCodec latent space. By integrating a time-conditioned 1D U‑Net and a Whisper-guided phoneme conditioning, the method achieves transcript-free purification with strong restoration of verifiability while preserving perceptual quality. Across diverse defenses, synthesis tools, and ASV back-ends, VocalBridge and its Whisper-enhanced variant consistently outperform baselines, and remain robust under cross-perturbation and adaptive attacks, underscoring the need for defenses that anticipate purification. The work contributes a thorough evaluation framework, a practical purification approach, and a call for more resilient protections against evolving voice-cloning threats with broad security and privacy implications.

Abstract

The rapid advancement of speech synthesis technologies, including text-to-speech (TTS) and voice conversion (VC), has intensified security and privacy concerns related to voice cloning. Recent defenses attempt to prevent unauthorized cloning by embedding protective perturbations into speech to obscure speaker identity while maintaining intelligibility. However, adversaries can apply advanced purification techniques to remove these perturbations, recover authentic acoustic characteristics, and regenerate cloneable voices. Despite the growing realism of such attacks, the robustness of existing defenses under adaptive purification remains insufficiently studied. Most existing purification methods are designed to counter adversarial noise in automatic speech recognition (ASR) systems rather than speaker verification or voice cloning pipelines. As a result, they fail to suppress the fine-grained acoustic cues that define speaker identity and are often ineffective against speaker verification attacks (SVA). To address these limitations, we propose Diffusion-Bridge (VocalBridge), a purification framework that learns a latent mapping from perturbed to clean speech in the EnCodec latent space. Using a time-conditioned 1D U-Net with a cosine noise schedule, the model enables efficient, transcript-free purification while preserving speaker-discriminative structure. We further introduce a Whisper-guided phoneme variant that incorporates lightweight temporal guidance without requiring ground-truth transcripts. Experimental results show that our approach consistently outperforms existing purification methods in recovering cloneable voices from protected speech. Our findings demonstrate the fragility of current perturbation-based defenses and highlight the need for more robust protection mechanisms against evolving voice-cloning and speaker verification threats.
Paper Structure (30 sections, 13 equations, 8 figures, 4 tables)

This paper contains 30 sections, 13 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Deepfake Speech Synthesis Techniques
  4. Speech Privacy and Protection Mechanisms
  5. Adversarial Purification
  6. Threat Model and Security Objectives
  7. Audio Diffusion Bridge Model
  8. Training Objective
  9. Inference
  10. Whisper-Guided Phoneme Conditioning
  11. Network Architecture
  12. Experiment and Evaluation
  13. Experimental Setup
  14. Datasets
  15. Protection Tools
  16. ...and 15 more sections

Figures (8)

  • Figure 1: This figure illustrates the threat model, demonstrating how an attacker leverages VocalBridge to bypass existing defenses and execute voice-cloning attacks.
  • Figure 2: VocalBridge Training
  • Figure 3: The mean ARR (%) over three speaker verification back-ends (x-vector, ECAPA-TDNN, and d-vector).
  • Figure 4: The spectrogram comparison shows that VocalBridge-W better maintains the structure and detail of the clean samples than the leading baseline De-AntiFake.
  • Figure 5: ARR under different adaptive strategies.
  • ...and 3 more figures