Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Erasing Your Voice Before It's Heard: Training-free Speaker Unlearning for Zero-shot Text-to-Speech

Myungjin Lee, Eunji Shin, Jiyoung Lee

TL;DR

Zero-shot TTS models pose privacy risks by enabling synthesis of non-consenting individuals’ voices. We introduce TruS, a training-free, inference-time framework that suppresses target speaker identities by steering identity-specific activations using an ID-prototype built from retain speakers. It dynamically selects identity-intervention layers and applies steered perturbations during generation to erase opt-out voices without retraining, generalizing to unseen speakers. Experiments on Emilia, LibriSpeech, and CREMA-D show significant identity suppression with preserved linguistic and emotional content, offering a scalable privacy safeguard for speech synthesis.

Abstract

Modern zero-shot text-to-speech (TTS) models offer unprecedented expressivity but also pose serious crime risks, as they can synthesize voices of individuals who never consented. In this context, speaker unlearning aims to prevent the generation of specific speaker identities upon request. Existing approaches, reliant on retraining, are costly and limited to speakers seen in the training set. We present TruS, a training-free speaker unlearning framework that shifts the paradigm from data deletion to inference-time control. TruS steers identity-specific hidden activations to suppress target speakers while preserving other attributes (e.g., prosody and emotion). Experimental results show that TruS effectively prevents voice generation on both seen and unseen opt-out speakers, establishing a scalable safeguard for speech synthesis. The demo and code are available on http://mmai.ewha.ac.kr/trus.

Erasing Your Voice Before It's Heard: Training-free Speaker Unlearning for Zero-shot Text-to-Speech

TL;DR

Zero-shot TTS models pose privacy risks by enabling synthesis of non-consenting individuals’ voices. We introduce TruS, a training-free, inference-time framework that suppresses target speaker identities by steering identity-specific activations using an ID-prototype built from retain speakers. It dynamically selects identity-intervention layers and applies steered perturbations during generation to erase opt-out voices without retraining, generalizing to unseen speakers. Experiments on Emilia, LibriSpeech, and CREMA-D show significant identity suppression with preserved linguistic and emotional content, offering a scalable privacy safeguard for speech synthesis.

Abstract

Modern zero-shot text-to-speech (TTS) models offer unprecedented expressivity but also pose serious crime risks, as they can synthesize voices of individuals who never consented. In this context, speaker unlearning aims to prevent the generation of specific speaker identities upon request. Existing approaches, reliant on retraining, are costly and limited to speakers seen in the training set. We present TruS, a training-free speaker unlearning framework that shifts the paradigm from data deletion to inference-time control. TruS steers identity-specific hidden activations to suppress target speakers while preserving other attributes (e.g., prosody and emotion). Experimental results show that TruS effectively prevents voice generation on both seen and unseen opt-out speakers, establishing a scalable safeguard for speech synthesis. The demo and code are available on http://mmai.ewha.ac.kr/trus.
Paper Structure (11 sections, 6 equations, 3 figures, 5 tables)

This paper contains 11 sections, 6 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Motivation and problem formulation
  4. Identity-specific steering vector
  5. Dynamic selection of steering layers
  6. Unlearning via inference-time steering
  7. Experiments
  8. Experimental settings
  9. Results
  10. Ablation study
  11. Conclusion

Figures (3)

  • Figure 1: Illustration of training-free speaker unlearning.
  • Figure 2: The overall framework of TruS, working with TTS models at inference time. Feature activations at layers and generation steps are optionally steered based on the dynamically selective threshold. With only a single utterance example of a target who requests to opt out, our method controls to suppress the identity-related activations without additional training.
  • Figure 3: Examples of step-wise cosine similarities between hidden activations of target speaker and ID-prototype, at 1st, 12th, 20th layers. Similarity increases in later steps at later layer, and decreases in earlier ones, indicating the need for dynamic layer- and step-specific steering.