VoxGuard: Evaluating User and Attribute Privacy in Speech via Membership Inference Attacks

TL;DR

VoxGuard reframes speech privacy as a membership inference problem and introduces two DP-inspired notions, User Privacy and Attribute Privacy, evaluated at low-FPR to reveal worst-case leakage. By analyzing synthetic and real data with both pretrained and fine-tuned adversaries, the study shows that privacy leakage can be orders of magnitude stronger at low-FPR than what EER indicates, and that speaker attributes such as gender and accent remain vulnerable even after anonymization. The framework demonstrates that simple, transparent attacks can achieve near-perfect attribute leakage, underscoring the need for explicit defenses and DP-aligned evaluation. VoxGuard thus provides a rigorous, low-FPR benchmark for assessing privacy leakage in voice anonymization systems and motivates defenses that ensure indistinguishability under realistic, high-risk attack scenarios.

Abstract

Voice anonymization aims to conceal speaker identity and attributes while preserving intelligibility, but current evaluations rely almost exclusively on Equal Error Rate (EER) that obscures whether adversaries can mount high-precision attacks. We argue that privacy should instead be evaluated in the low false-positive rate (FPR) regime, where even a small number of successful identifications constitutes a meaningful breach. To this end, we introduce VoxGuard, a framework grounded in differential privacy and membership inference that formalizes two complementary notions: User Privacy, preventing speaker re-identification, and Attribute Privacy, protecting sensitive traits such as gender and accent. Across synthetic and real datasets, we find that informed adversaries, especially those using fine-tuned models and max-similarity scoring, achieve orders-of-magnitude stronger attacks at low-FPR despite similar EER. For attributes, we show that simple transparent attacks recover gender and accent with near-perfect accuracy even after anonymization. Our results demonstrate that EER substantially underestimates leakage, highlighting the need for low-FPR evaluation, and recommend VoxGuard as a benchmark for evaluating privacy leakage.

Figures (2)

• Figure 1: Illustration of the four VoxGuard evaluation tasks. User-Privacy-Strict: adversary compares same-text pairs to decide if two samples belong to the same speaker (identity as the varying factor). User-Privacy-Relaxed: adversary compares arbitrary texts from two speakers, reflecting the general case where content varies. Attr-Privacy-Strict: pairs are matched on all attributes except one, isolating the effect of a single attribute. Attr-Privacy-Relaxed: pairs differ in one attribute without controlling for others, modeling the general case where multiple traits vary.
• Figure 2: Comparing ROC curves in log–log scale reveals severe gaps between User Privacy attacks. While average-case EERs range only from 20–47%, their performance at low FPR differs drastically. FT–Max at $L=200$ achieves TPR $>$50% at $\text{FPR}=10^{-4}$, whereas PT–Avg remains near random. This shows that reporting only EER obscures strong privacy leaks. Max scoring and fine-tuned attackers yield orders-of-magnitude stronger attacks in the low-FPR regime.

Theorems & Definitions (3)

• Definition 1: ($\epsilon, \delta$)-User Privacy
• Definition 2: Strict ($\epsilon$,$\delta$)-Attribute Privacy
• Definition 3: Relaxed ($\epsilon$,$\delta$)-Attribute Privacy