Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Causally Disentangled Contrastive Learning for Multilingual Speaker Embeddings

Mariëtte Olijslager, Seyed Sahand Mohammadi Ziabari, Ali Mohammed Mansoor Alsahag

TL;DR

This work investigates demographic information leakage in self-supervised speaker embeddings trained with SimCLR, focusing on gender, age, and accent. It compares adversarial debiasing and a causal bottleneck architecture, using both linear and nonlinear probes to quantify leakage and evaluating speaker verification with ROC-AUC and EER. The findings show gender is strongly and linearly encoded, age and accent are weaker and often nonlinear; adversarial debiasing reduces gender leakage but harms verification, while the causal bottleneck better suppresses leakage at a substantial verification cost. The results reveal fundamental trade-offs and suggest that effective, domain-general debiasing remains challenging, with nonlinear leakage and cross-domain variability limiting current approaches.

Abstract

Self-supervised speaker embeddings are widely used in speaker verification systems, but prior work has shown that they often encode sensitive demographic attributes, raising fairness and privacy concerns. This paper investigates the extent to which demographic information, specifically gender, age, and accent, is present in SimCLR-trained speaker embeddings and whether such leakage can be mitigated without severely degrading speaker verification performance. We study two debiasing strategies: adversarial training through gradient reversal and a causal bottleneck architecture that explicitly separates demographic and residual information. Demographic leakage is quantified using both linear and nonlinear probing classifiers, while speaker verification performance is evaluated using ROC-AUC and EER. Our results show that gender information is strongly and linearly encoded in baseline embeddings, whereas age and accent are weaker and primarily nonlinearly represented. Adversarial debiasing reduces gender leakage but has limited effect on age and accent and introduces a clear trade-off with verification accuracy. The causal bottleneck further suppresses demographic information, particularly in the residual representation, but incurs substantial performance degradation. These findings highlight fundamental limitations in mitigating demographic leakage in self-supervised speaker embeddings and clarify the trade-offs inherent in current debiasing approaches.

Causally Disentangled Contrastive Learning for Multilingual Speaker Embeddings

TL;DR

This work investigates demographic information leakage in self-supervised speaker embeddings trained with SimCLR, focusing on gender, age, and accent. It compares adversarial debiasing and a causal bottleneck architecture, using both linear and nonlinear probes to quantify leakage and evaluating speaker verification with ROC-AUC and EER. The findings show gender is strongly and linearly encoded, age and accent are weaker and often nonlinear; adversarial debiasing reduces gender leakage but harms verification, while the causal bottleneck better suppresses leakage at a substantial verification cost. The results reveal fundamental trade-offs and suggest that effective, domain-general debiasing remains challenging, with nonlinear leakage and cross-domain variability limiting current approaches.

Abstract

Self-supervised speaker embeddings are widely used in speaker verification systems, but prior work has shown that they often encode sensitive demographic attributes, raising fairness and privacy concerns. This paper investigates the extent to which demographic information, specifically gender, age, and accent, is present in SimCLR-trained speaker embeddings and whether such leakage can be mitigated without severely degrading speaker verification performance. We study two debiasing strategies: adversarial training through gradient reversal and a causal bottleneck architecture that explicitly separates demographic and residual information. Demographic leakage is quantified using both linear and nonlinear probing classifiers, while speaker verification performance is evaluated using ROC-AUC and EER. Our results show that gender information is strongly and linearly encoded in baseline embeddings, whereas age and accent are weaker and primarily nonlinearly represented. Adversarial debiasing reduces gender leakage but has limited effect on age and accent and introduces a clear trade-off with verification accuracy. The causal bottleneck further suppresses demographic information, particularly in the residual representation, but incurs substantial performance degradation. These findings highlight fundamental limitations in mitigating demographic leakage in self-supervised speaker embeddings and clarify the trade-offs inherent in current debiasing approaches.
Paper Structure (27 sections, 1 figure, 11 tables)

This paper contains 27 sections, 1 figure, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Demographic information in speaker embeddings
  4. Contrastive learning methods for speaker embeddings
  5. Causal disentanglement
  6. Debiasing representations via adversarial debiasing and bottlenecks
  7. Methodology
  8. Datasets
  9. Dataset attributes
  10. Preprocessing
  11. Speaker distribution and splits
  12. Audioprocessing
  13. Training pipeline
  14. SimCLR baseline
  15. Adversarial debiasing
  16. ...and 12 more sections

Figures (1)

  • Figure 1: Directed acyclic graph (DAG) of the causal bottleneck layer architecture. The diagram illustrates how the model explicitly separates speaker-discriminative information from demographic factors (gender, age, accent) by enforcing a causal structure in the embedding space.