RoVo: Robust Voice Protection Against Unauthorized Speech Synthesis with Embedding-Level Perturbations

TL;DR

RoVo tackles the threat of unauthorized speech synthesis by injecting adversarial perturbations into high-dimensional voice embeddings rather than the audio signal, enabling perturbations to survive common post-processing. It employs a NAC/BARK-based backbone with coarse and fine Transformers and optimizes perturbations using a PerC-AL loss that alternates between disrupting speaker embeddings and preserving perceptual quality. Across multiple synthesis and enhancement models, RoVo delivers large Defense Success Rate gains and strong robustness to speech enhancement, including near-perfect protection on a commercial API, with user studies showing preserved naturalness. The work demonstrates a practical, proactive defense framework that significantly improves protection against voice cloning and related abuses while maintaining usability.

Abstract

With the advancement of AI-based speech synthesis technologies such as Deep Voice, there is an increasing risk of voice spoofing attacks, including voice phishing and fake news, through unauthorized use of others' voices. Existing defenses that inject adversarial perturbations directly into audio signals have limited effectiveness, as these perturbations can easily be neutralized by speech enhancement methods. To overcome this limitation, we propose RoVo (Robust Voice), a novel proactive defense technique that injects adversarial perturbations into high-dimensional embedding vectors of audio signals, reconstructing them into protected speech. This approach effectively defends against speech synthesis attacks and also provides strong resistance to speech enhancement models, which represent a secondary attack threat. In extensive experiments, RoVo increased the Defense Success Rate (DSR) by over 70% compared to unprotected speech, across four state-of-the-art speech synthesis models. Specifically, RoVo achieved a DSR of 99.5% on a commercial speaker-verification API, effectively neutralizing speech synthesis attack. Moreover, RoVo's perturbations remained robust even under strong speech enhancement conditions, outperforming traditional methods. A user study confirmed that RoVo preserves both naturalness and usability of protected speech, highlighting its effectiveness in complex and evolving threat scenarios.

Figures (5)

• Figure 1: Overview of RoVo: While signal-level protected audio can defend against the initial attack, its effectiveness significantly decreases under realistic secondary post-processing attacks (e.g., speech enhancement). In contrast, RoVo maintains robust defense even against such post-processing, ultimately influencing whether the victim is Deceived or Not Deceived.
• Figure 2: Architecture of speech synthesis
• Figure 3: Overview of RoVo framework: RoVo injects adversarial perturbations directly into embedding vectors extracted by the Neural Codec Encoder, effectively disrupting speaker-specific features and reconstructing protected audio through the Neural Codec Decoder.
• Figure 4: Spectrogram comparison: showing Antifake and RoVo before and after speech enhancement. RoVo perturbations remain robust, while Antifake perturbations are easily removed.
• Figure 5: User Study Results: Comparison of Perceived Similarity Before and After Speech Enhancement.