VoiceCloak: A Multi-Dimensional Defense Framework against Unauthorized Diffusion-based Voice Cloning

TL;DR

VoiceCloak tackles the rising risk of unauthorized diffusion-based voice cloning by introducing a proactive, multi-dimensional defense that targets diffusion-specific vulnerabilities. The framework combines four adversarial objectives—opposite-gender embedding guidance, attention-context divergence, score-magnitude amplification, and semantic corruption of U-Net features—into a joint optimization that perturbs reference audio within a small, inaudible budget. Empirical results on LibriTTS and VCTK show substantial defense success against diffusion-based VC, with strong identity obfuscation, degraded output quality, transferability to unseen models, and robustness to common post-processing. The work offers a practical offline protection approach, requiring modest hardware and providing clear metrics for practitioners seeking to safeguard voice data before dissemination.

Abstract

Diffusion Models (DMs) have achieved remarkable success in realistic voice cloning (VC), while they also increase the risk of malicious misuse. Existing proactive defenses designed for traditional VC models aim to disrupt the forgery process, but they have been proven incompatible with DMs due to the intricate generative mechanisms of diffusion. To bridge this gap, we introduce VoiceCloak, a multi-dimensional proactive defense framework with the goal of obfuscating speaker identity and degrading perceptual quality in potential unauthorized VC. To achieve these goals, we conduct a focused analysis to identify specific vulnerabilities within DMs, allowing VoiceCloak to disrupt the cloning process by introducing adversarial perturbations into the reference audio. Specifically, to obfuscate speaker identity, VoiceCloak first targets speaker identity by distorting representation learning embeddings to maximize identity variation, which is guided by auditory perception principles. Additionally, VoiceCloak disrupts crucial conditional guidance processes, particularly attention context, thereby preventing the alignment of vocal characteristics that are essential for achieving convincing cloning. Then, to address the second objective, VoiceCloak introduces score magnitude amplification to actively steer the reverse trajectory away from the generation of high-quality speech. Noise-guided semantic corruption is further employed to disrupt structural speech semantics captured by DMs, degrading output quality. Extensive experiments highlight VoiceCloak's outstanding defense success rate against unauthorized diffusion-based voice cloning. Audio samples of VoiceCloak are available at https://voice-cloak.github.io/VoiceCloak/.

Figures (6)

• Figure 1: Illustration of diffusion-based voice cloning malicious misuse. (a) Voice forgery enables threats of fraud. (b) Traditional methods struggle due to ineffective disruptive gradients. (c) Audio protected by VoiceCloak resists high-fidelity cloning.
• Figure 2: Overview of the proposed framework. Perturbation optimization is guided by gradients from $\mathcal{L}_{total}$, aggregating four targeting two objectives: (1) Identity Obfuscation (via Opposite-Gender Centroid Guidance and Attention Context Divergence) and (2) Perceptual Fidelity Degradation (Score Magnitude Amplification and Noise-Guided Semantic Corruption).
• Figure 3: Mel spectrograms with $F_0$ pitch contours (green lines), and inferred intonation of the corresponding words. Arrows indicate perceived intonation shifts. (Intonation aligns with the ground truth, which is marked by green arrows, and diverges, which is marked by red arrows.)
• Figure 4: Protecting commercial speaker verification APIs (Iflytek, Azure) from spoofing attacks (lower are better).
• Figure 5: User perceptual study results. (a) Timbre Dissimilarity Preference. (b) Corresponding results for perceived Naturalness Disruption.