WavShape: Information-Theoretic Speech Representation Learning for Fair and Privacy-Aware Audio Processing

TL;DR

WavShape is proposed, an information-theoretic speech representation learning framework that optimizes embeddings for fairness and privacy while preserving task-relevant information in speech embeddings.

Abstract

Speech embeddings often retain sensitive attributes such as speaker identity, accent, or demographic information, posing risks in biased model training and privacy leakage. We propose WavShape, an information-theoretic speech representation learning framework that optimizes embeddings for fairness and privacy while preserving task-relevant information. We leverage mutual information (MI) estimation using the Donsker-Varadhan formulation to guide an MI-based encoder that systematically filters sensitive attributes while maintaining speech content essential for downstream tasks. Experimental results on three known datasets show that WavShape reduces MI between embeddings and sensitive attributes by up to 81% while retaining 97% of task-relevant information. By integrating information theory with self-supervised speech models, this work advances the development of fair, privacy-aware, and resource-efficient speech systems.

Figures (4)

• Figure 1: How does WavShape encode fair and bias-free speech representations? Our framework first extracts speech embeddings using a pre-trained speech encoder (e.g., Whisper). These embeddings, which may contain private and biased attributes, are then transformed via the WavShape projection layer to generate bias-mitigated public embeddings. The IT evaluator (blue) optimizes WavShape by minimizing the MI between speech embeddings and sensitive attributes while maximizing MI with task-relevant features and original representation. During training, the feature extractor is frozen, only WavShape and the IT evaluator are updated. At inference, the IT evaluator is removed, and WavShape is used in conjunction with the base feature extractor (green), ensuring bias-free and privacy-preserving embeddings for fair and inclusive modeling.
• Figure 2: We train two pairs of classifiers to predict task-relevant (left) and sensitive (right) binary labels (yellow and purple). We train one pair with Whisper embeddings while the other gets our embeddings as input. We project the whole CV dataset into a 2D t-SNE space with the trained classifiers and visualize the best decision boundary. The task-relevant structure remains intact while sensitive attributes are effectively obscured compared to the models trained with original embeddings.
• Figure 3: ROC and AUROC scores for the classifiers trained to predict task-relevant attributes (left) and sensitive attributes (right). The results demonstrate that our MI-based encoding preserves task-relevant distinctions while effectively reducing the predictability of sensitive attributes.
• Figure 4: IT Estimator's mean MI values for task-related $T_i(.)$ (utility) and sensitive $S_i(.)$ (privacy) features over the number of epochs. WS embeddings get more descriptive for the tasks while suppressing sensitive information over time.