On the Generation and Removal of Speaker Adversarial Perturbation for Voice-Privacy Protection

TL;DR

The paper investigates the reversibility of speaker adversarial perturbations for voice privacy by proposing a joint-training framework that learns a perturbation generator and a removal module using a symmetric saliency-based encoder-decoder (SSED). It formalizes the generation of perturbations $\boldsymbol{\delta}=\epsilon\cdot(\boldsymbol{n}\odot\boldsymbol{m})$ and the restoration of original speech $\hat{\boldsymbol{x}}$ from adversarial input, optimizing both speaker-embedding decorrelation and speech quality. Through LibriSpeech experiments with ECAPA-TDNN embeddings and Whisper ASR, the approach achieves restoration of speaker attributes and content, outperforming purification baselines that fail to fully remove perturbations. The findings demonstrate practical potential for controlled, reversible voice-privacy protections in forensic and security contexts, while suggesting directions for generalization to out-of-domain data and denoising-based strategies.

Abstract

Neural networks are commonly known to be vulnerable to adversarial attacks mounted through subtle perturbation on the input data. Recent development in voice-privacy protection has shown the positive use cases of the same technique to conceal speaker's voice attribute with additive perturbation signal generated by an adversarial network. This paper examines the reversibility property where an entity generating the adversarial perturbations is authorized to remove them and restore original speech (e.g., the speaker him/herself). A similar technique could also be used by an investigator to deanonymize a voice-protected speech to restore criminals' identities in security and forensic analysis. In this setting, the perturbation generative module is assumed to be known in the removal process. To this end, a joint training of perturbation generation and removal modules is proposed. Experimental results on the LibriSpeech dataset demonstrated that the subtle perturbations added to the original speech can be predicted from the anonymized speech while achieving the goal of privacy protection. By removing these perturbations from the anonymized sample, the original speech can be restored. Audio samples can be found in \url{https://voiceprivacy.github.io/Perturbation-Generation-Removal/}.

Figures (3)

• Figure 1: The process of voice-privacy protection and restoration.
• Figure 2: Framework of symmetric saliency-based encoder-decoder (SSED). The noise and mask are generated by the noise&mask generator, represented in the rectangular box of the blue dotted line. The red lines are applicable only in training, while the black lines are valid in both.
• Figure 3: Model architecture for speaker adversarial perturbation generation and removal. The rectangular boxes of the red and blue dotted lines contain the perturbation generator and the removal module, respectively. The latter takes the adversarial sample ${\boldsymbol{x}}'$ generated by the former. The noise&mask generator module used in the perturbation generator is inherited from Fig \ref{['fig. SSED']}. The red and blue lines are applicable only in training and inference, respectively, while the black lines are valid in both.