Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Privacy-Preserving End-to-End Full-Duplex Speech Dialogue Models

Nikita Kuzmin, Tao Zhong, Jiajun Deng, Yingke Zhu, Tristan Tsoi, Tianxiang Cao, Simon Lui, Kong Aik Lee, Eng Siong Chng

TL;DR

It is shown that the hidden states of SALM-Duplex and Moshi leak substantial speaker identity across all transformer layers, with SALM-Duplex showing stronger leakage in early layers while Moshi leaks uniformly, and that Linkability rises sharply within the first few turns.

Abstract

End-to-end full-duplex speech models feed user audio through an always-on LLM backbone, yet the speaker privacy implications of their hidden representations remain unexamined. Following the VoicePrivacy 2024 protocol with a lazy-informed attacker, we show that the hidden states of SALM-Duplex and Moshi leak substantial speaker identity across all transformer layers. Layer-wise and turn-wise analyses reveal that leakage persists across all layers, with SALM-Duplex showing stronger leakage in early layers while Moshi leaks uniformly, and that Linkability rises sharply within the first few turns. We propose two streaming anonymization setups using Stream-Voice-Anon: a waveform-level front-end (Anon-W2W) and a feature-domain replacement (Anon-W2F). Anon-W2F raises EER by over 3.5x relative to the discrete encoder baseline (11.2% to 41.0%), approaching the 50% random-chance ceiling, while Anon-W2W retains 78-93% of baseline sBERT across setups with sub-second response latency (FRL under 0.8 s).

Privacy-Preserving End-to-End Full-Duplex Speech Dialogue Models

TL;DR

It is shown that the hidden states of SALM-Duplex and Moshi leak substantial speaker identity across all transformer layers, with SALM-Duplex showing stronger leakage in early layers while Moshi leaks uniformly, and that Linkability rises sharply within the first few turns.

Abstract

End-to-end full-duplex speech models feed user audio through an always-on LLM backbone, yet the speaker privacy implications of their hidden representations remain unexamined. Following the VoicePrivacy 2024 protocol with a lazy-informed attacker, we show that the hidden states of SALM-Duplex and Moshi leak substantial speaker identity across all transformer layers. Layer-wise and turn-wise analyses reveal that leakage persists across all layers, with SALM-Duplex showing stronger leakage in early layers while Moshi leaks uniformly, and that Linkability rises sharply within the first few turns. We propose two streaming anonymization setups using Stream-Voice-Anon: a waveform-level front-end (Anon-W2W) and a feature-domain replacement (Anon-W2F). Anon-W2F raises EER by over 3.5x relative to the discrete encoder baseline (11.2% to 41.0%), approaching the 50% random-chance ceiling, while Anon-W2W retains 78-93% of baseline sBERT across setups with sub-second response latency (FRL under 0.8 s).
Paper Structure (17 sections, 2 figures, 2 tables)

This paper contains 17 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. E2E speech dialogue language models
  4. Speaker anonymization
  5. Privacy in neural speech and language representations
  6. Methodology
  7. E2E full-duplex dialogue models
  8. User encoder variants
  9. Anonymization setups
  10. Anon-W2W: Wave-to-wave anonymization
  11. Anon-W2F: Wave-to-feature anonymization
  12. Hidden state extraction
  13. Setup
  14. Results
  15. Ablation studies
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Speaker privacy ($1 - \mathrm{Linkability}$) vs. dialogue turn count. Only discrete encoder variants are shown; continuous encoder Linkability is omitted for clarity. Red lines: no anonymization; green lines: post-anonymization. Without anonymization, both systems drop into the low-privacy zone within a few turns; anonymization lifts privacy into the protected zone, though Moshi + W2W shows gradual degradation over dialogue length.
  • Figure 2: Overview of the original SALM-Duplex pipeline and proposed anonymization setups. The main diagram shows the ASR-based encoder baseline: an ECAPA-TDNN probe attached to the LLM's hidden states (red dashed path) reveals substantial speaker identity leakage. The Anon-W2W inset (upper left) prepends Stream-Voice-Anon to anonymize the waveform before the unchanged ASR encoder. The Anon-W2F inset (upper right) replaces the ASR encoder with the Stream-Voice-Anon encoder (anonymization active) and fine-tunes the LLM, eliminating the redundant waveform synthesis step. (Anon-W2F is demonstrated for SALM-Duplex; the Anon-W2W setup is additionally evaluated on Moshi.)