Universal Semantic Disentangled Privacy-preserving Speech Representation Learning

TL;DR

The paper addresses privacy concerns in training speech-enabled models by introducing the Universal Speech Codec (USC), a two-branch encoder–decoder that disentangles semantics from speaker identity. The semantic branch captures content and paralinguistics while a residual branch preserves speaker information for high-fidelity reconstruction, with privacy controls including speaker reversal, semantic distillation, quantizer dropout, and local differential privacy. A novel privacy evaluation framework based on $k$-anonymity, linking privacy metrics to perceptual tests, demonstrates that USC substantially obscures speaker identity while maintaining useful semantic information and prosody. Results show competitive reconstruction quality at ultra-low bitrates and meaningful privacy improvements, especially when applying LDP, highlighting a practical trade-off for secure, speech-aware LLMs. The work also discusses ethical considerations and potential voice-conversion capabilities enabled by the learned disentangled representations.

Abstract

The use of audio recordings of human speech to train LLMs poses privacy concerns due to these models' potential to generate outputs that closely resemble artifacts in the training data. In this study, we propose a speaker privacy-preserving representation learning method through the Universal Speech Codec (USC), a computationally efficient encoder-decoder model that disentangles speech into: (i) privacy-preserving semantically rich representations, capturing content and speech paralinguistics, and (ii) residual acoustic and speaker representations that enables high-fidelity reconstruction. Extensive evaluations presented show that USC's semantic representation preserves content, prosody, and sentiment, while removing potentially identifiable speaker attributes. Combining both representations, USC achieves state-of-the-art speech reconstruction. Additionally, we introduce an evaluation methodology for measuring privacy-preserving properties, aligning with perceptual tests. We compare USC against other codecs in the literature and demonstrate its effectiveness on privacy-preserving representation learning, illustrating the trade-offs of speaker anonymization, paralinguistics retention and content preservation in the learned semantic representations. Audio samples are shared in https://www.amazon.science/usc-samples.

Figures (9)

• Figure 1: General scheme for privacy-preserving speech representation learning and generative modeling. Dashed lines denote feature extraction. Solid lines shows the privacy-preserving modelling.
• Figure 2: USC architecture. Red dashed lines denote training objectives while black continuous lines refer to the inference pipeline for high-fidelity and semantic reconstruction of speech.
• Figure 3: Spectrogram visualization of semantic $C_0$ reconstruction of a speech sample from all the compared baselines, with a zoomed in view of the pitch harmonics. More examples in Appendix \ref{['app:semantic_melspecs']}.
• Figure 4: Voice Conversion pipeline for a LLM-based TTS model trained to predict USC representations. Semantics is extracted from source speaker (blue) and identity from target speaker (yellow).
• Figure 5: Mel-Sepctrogram visualization of VC capabilities through semantic $C_0$ partial-teacher forcing (PTF) without text: a source low-pitched speech converted to a target high-pitch speaker.