Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stream-Voice-Anon: Enhancing Utility of Real-Time Speaker Anonymization via Neural Audio Codec and Language Models

Nikita Kuzmin, Songting Liu, Kong Aik Lee, Eng Siong Chng

TL;DR

Stream-Voice-Anon addresses the challenge of real-time speaker anonymization by uniting neural audio codec representations with causal language model conditioning in a streaming framework. It introduces pseudo-speaker sampling, prompt-based conditioning, and inference-time anonymization to maintain intelligibility and emotion while protecting privacy under latency constraints. Across VoicePrivacy 2024 benchmarks, the approach delivers up to 46% relative improvements in WER and up to 28% in UAR over the prior streaming SOTA, with comparable latency and robust privacy against lazy-informed attackers, though some vulnerability remains against semi-informed attackers. The work demonstrates a practical, tunable path for deploying real-time anonymization in live applications and outlines future directions toward CPU deployment and stronger defenses against adaptive threats.

Abstract

Protecting speaker identity is crucial for online voice applications, yet streaming speaker anonymization (SA) remains underexplored. Recent research has demonstrated that neural audio codec (NAC) provides superior speaker feature disentanglement and linguistic fidelity. NAC can also be used with causal language models (LM) to enhance linguistic fidelity and prompt control for streaming tasks. However, existing NAC-based online LM systems are designed for voice conversion (VC) rather than anonymization, lacking the techniques required for privacy protection. Building on these advances, we present Stream-Voice-Anon, which adapts modern causal LM-based NAC architectures specifically for streaming SA by integrating anonymization techniques. Our anonymization approach incorporates pseudo-speaker representation sampling, a speaker embedding mixing and diverse prompt selection strategies for LM conditioning that leverage the disentanglement properties of quantized content codes to prevent speaker information leakage. Additionally, we compare dynamic and fixed delay configurations to explore latency-privacy trade-offs in real-time scenarios. Under the VoicePrivacy 2024 Challenge protocol, Stream-Voice-Anon achieves substantial improvements in intelligibility (up to 46% relative WER reduction) and emotion preservation (up to 28% UAR relative) compared to the previous state-of-the-art streaming method DarkStream while maintaining comparable latency (180ms vs 200ms) and privacy protection against lazy-informed attackers, though showing 15% relative degradation against semi-informed attackers.

Stream-Voice-Anon: Enhancing Utility of Real-Time Speaker Anonymization via Neural Audio Codec and Language Models

TL;DR

Stream-Voice-Anon addresses the challenge of real-time speaker anonymization by uniting neural audio codec representations with causal language model conditioning in a streaming framework. It introduces pseudo-speaker sampling, prompt-based conditioning, and inference-time anonymization to maintain intelligibility and emotion while protecting privacy under latency constraints. Across VoicePrivacy 2024 benchmarks, the approach delivers up to 46% relative improvements in WER and up to 28% in UAR over the prior streaming SOTA, with comparable latency and robust privacy against lazy-informed attackers, though some vulnerability remains against semi-informed attackers. The work demonstrates a practical, tunable path for deploying real-time anonymization in live applications and outlines future directions toward CPU deployment and stronger defenses against adaptive threats.

Abstract

Protecting speaker identity is crucial for online voice applications, yet streaming speaker anonymization (SA) remains underexplored. Recent research has demonstrated that neural audio codec (NAC) provides superior speaker feature disentanglement and linguistic fidelity. NAC can also be used with causal language models (LM) to enhance linguistic fidelity and prompt control for streaming tasks. However, existing NAC-based online LM systems are designed for voice conversion (VC) rather than anonymization, lacking the techniques required for privacy protection. Building on these advances, we present Stream-Voice-Anon, which adapts modern causal LM-based NAC architectures specifically for streaming SA by integrating anonymization techniques. Our anonymization approach incorporates pseudo-speaker representation sampling, a speaker embedding mixing and diverse prompt selection strategies for LM conditioning that leverage the disentanglement properties of quantized content codes to prevent speaker information leakage. Additionally, we compare dynamic and fixed delay configurations to explore latency-privacy trade-offs in real-time scenarios. Under the VoicePrivacy 2024 Challenge protocol, Stream-Voice-Anon achieves substantial improvements in intelligibility (up to 46% relative WER reduction) and emotion preservation (up to 28% UAR relative) compared to the previous state-of-the-art streaming method DarkStream while maintaining comparable latency (180ms vs 200ms) and privacy protection against lazy-informed attackers, though showing 15% relative degradation against semi-informed attackers.
Paper Structure (18 sections, 5 equations, 2 figures, 3 tables)

This paper contains 18 sections, 5 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Proposed Approach
  3. Overview
  4. Content encoder
  5. Acoustic encoder
  6. Speaker encoder
  7. Autoregressive Voice Conversion (ARVC)
  8. Inference-time anonymization techniques
  9. Experimental Setup
  10. Datasets
  11. Evaluation Protocol
  12. Model Configurations
  13. Results and Discussion
  14. Model comparison
  15. Prompt diversity
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Training and inference pipelines of the Stream-Voice-Anon.
  • Figure 2: Latency (x-axis) against privacy (EER $\uparrow$, left y-axis) and intelligibility (WER $\downarrow$, right y-axis) for Stream-Voice-Anon with $d\in\{1, 2, 4, 8\}$ dynamic delay values. Bold $\pmb{\times}$, $\pmb{\times}$ markers indicate a fixed-delay model with $d=4$ for comparison. Selection strategy is vctk-1rnd.