Adversarial speech for voice privacy protection from Personalized Speech generation

TL;DR

The paper addresses the risk of misuse of personalized TTS/VC systems by proposing a proactive voice privacy method that perturbs input speech to block downstream speaker encoding. It attacks the speaker encoder within a white-box YourTTS setup using FGSM and particularly I-FGSM on STFT-based features to maximize cosine-distance between original and perturbed embeddings. Evaluations on LibriSpeech indicate that I-FGSM perturbations yield higher ASV EERs for both TTS and VC references while maintaining audio quality better than simple Gaussian noise. This work demonstrates a practical approach to mitigating impersonation risks in personalized speech generation and informs future defenses against voice privacy threats.

Abstract

The rapid progress in personalized speech generation technology, including personalized text-to-speech (TTS) and voice conversion (VC), poses a challenge in distinguishing between generated and real speech for human listeners, resulting in an urgent demand in protecting speakers' voices from malicious misuse. In this regard, we propose a speaker protection method based on adversarial attacks. The proposed method perturbs speech signals by minimally altering the original speech while rendering downstream speech generation models unable to accurately generate the voice of the target speaker. For validation, we employ the open-source pre-trained YourTTS model for speech generation and protect the target speaker's speech in the white-box scenario. Automatic speaker verification (ASV) evaluations were carried out on the generated speech as the assessment of the voice protection capability. Our experimental results show that we successfully perturbed the speaker encoder of the YourTTS model using the gradient-based I-FGSM adversarial perturbation method. Furthermore, the adversarial perturbation is effective in preventing the YourTTS model from generating the speech of the target speaker. Audio samples can be found in https://voiceprivacy.github.io/Adeversarial-Speech-with-YourTTS.

Figures (3)

• Figure 1: The inference flowchart of the YourTTS model. The blue and red boxes represent the flows of TTS and VC, respectively.
• Figure 2: The flowchart of generating adversarial samples based on I-FGSM. The ${\bf x}$ denotes the STFT feature extracted from the original reference speech; $\delta_i$ and ${\bf \tilde{x}}_i$ represent the adversarial perturbation and the resultant adversarial STFT in the $i$-th iteration, respectively. When $i=0$, ${\bf \tilde{x}}_0$ is initialized with $\bf x$.
• Figure 3: The similarity among utterances (first row) and speakers (second row). The speaker similarity between the recording ($rec$) and the speech generated using the recording as the reference speech ($tts$ and $vc$), using adversarial speech for reference ($tts^*$ and $vc^*$) are presented, respectively.