Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DELULU: Discriminative Embedding Learning Using Latent Units for Speaker-Aware Self-Supervised Speech Foundational Model

Massa Baali, Rita Singh, Bhiksha Raj

TL;DR

DELULU introduces a speaker-aware self-supervised framework that embeds speaker identity into SSL by guiding clustering with frame-level embeddings from a state-of-the-art SV model, ReDimNet. It employs a dual objective of masked prediction and denoising, with a final loss $L_{total} = L_{mask} + \lambda L_{denoise}$, to learn robust, speaker-discriminative representations. Across upstream and downstream speaker-verification benchmarks, zero-shot profiling, and demographic analyses, DELULU yields significant gains (e.g., up to 62% relative EER improvement on VoxO1) and demonstrates strong generalization to profiling tasks like age and accent without task-specific supervision. This work highlights a scalable principle: incorporating targeted external supervision into pseudo-label formation can strengthen SSL foundations for domain-specific discriminability and broad downstream transfer.

Abstract

Self-supervised speech models have achieved remarkable success on content-driven tasks, yet they remain limited in capturing speaker-discriminative features critical for verification, diarization, and profiling applications. We introduce DELULU, a speaker-aware self-supervised foundational model that addresses this limitation by integrating external supervision into the pseudo-label generation process. DELULU leverages frame-level embeddings from ReDimNet, a state-of-the-art speaker verification model, to guide the k-means clustering step during pre-training, introducing a strong speaker-discriminative inductive bias that aligns representation learning with speaker identity. The model is trained using a dual objective that combines masked prediction and denoising, further enhancing robustness and generalization. DELULU significantly outperforms prior self-supervised learning (SSL) models across a range of speaker-centric tasks, achieving up to 62% relative improvement in equal error rate (EER) for speaker verification and consistent gains on zero-shot profiling tasks such as gender, age, accent, and speaker counting. Our findings demonstrate that DELULU is a strong universal encoder for speaker-aware speech processing, enabling superior performance even without task-specific fine-tuning.

DELULU: Discriminative Embedding Learning Using Latent Units for Speaker-Aware Self-Supervised Speech Foundational Model

TL;DR

DELULU introduces a speaker-aware self-supervised framework that embeds speaker identity into SSL by guiding clustering with frame-level embeddings from a state-of-the-art SV model, ReDimNet. It employs a dual objective of masked prediction and denoising, with a final loss , to learn robust, speaker-discriminative representations. Across upstream and downstream speaker-verification benchmarks, zero-shot profiling, and demographic analyses, DELULU yields significant gains (e.g., up to 62% relative EER improvement on VoxO1) and demonstrates strong generalization to profiling tasks like age and accent without task-specific supervision. This work highlights a scalable principle: incorporating targeted external supervision into pseudo-label formation can strengthen SSL foundations for domain-specific discriminability and broad downstream transfer.

Abstract

Self-supervised speech models have achieved remarkable success on content-driven tasks, yet they remain limited in capturing speaker-discriminative features critical for verification, diarization, and profiling applications. We introduce DELULU, a speaker-aware self-supervised foundational model that addresses this limitation by integrating external supervision into the pseudo-label generation process. DELULU leverages frame-level embeddings from ReDimNet, a state-of-the-art speaker verification model, to guide the k-means clustering step during pre-training, introducing a strong speaker-discriminative inductive bias that aligns representation learning with speaker identity. The model is trained using a dual objective that combines masked prediction and denoising, further enhancing robustness and generalization. DELULU significantly outperforms prior self-supervised learning (SSL) models across a range of speaker-centric tasks, achieving up to 62% relative improvement in equal error rate (EER) for speaker verification and consistent gains on zero-shot profiling tasks such as gender, age, accent, and speaker counting. Our findings demonstrate that DELULU is a strong universal encoder for speaker-aware speech processing, enabling superior performance even without task-specific fine-tuning.
Paper Structure (36 sections, 3 equations, 4 figures, 5 tables)

This paper contains 36 sections, 3 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Self-Supervised Speech Representation Learning
  4. Supervised Speaker Representation Learning
  5. Self-Supervised Learning for Speaker Tasks
  6. Robustness and Denoising in Speech Models
  7. Architecture
  8. DELULU Encoder
  9. Teacher-Guided Clustering
  10. Training
  11. Denoising Loss:
  12. Masked Prediction Loss:
  13. Experimental Setup
  14. Ablation Study
  15. Evaluation
  16. ...and 21 more sections

Figures (4)

  • Figure 1: Illustration of DELULU architecture. The student encoder is trained with (a) a denoising objective aligning clean and noisy features, and (b) a masked prediction objective, where frame-level features from the teacher (ReDimNet) are clustered into discrete speaker-aware targets for cross-entropy training.
  • Figure 2: t-SNE visualization of embeddings from 40 VoxO1 speakers. DELULU yields compact, well-separated clusters with lower intra-speaker variability and higher inter-speaker separation than baseline SSL models.
  • Figure 3: t-SNE visualization of speaker embeddings by gender on the EARS dataset. DELULU shows clear speaker clusters and strong gender separation.
  • Figure 4: t-SNE visualization of EARS embeddings by gender and age groups (Table \ref{['tab:demographic_eer']}). DELULU clusters primarily by speaker identity, with demographic traits forming secondary structure and no apparent bias.