Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DAME: Duration-Aware Matryoshka Embedding for Duration-Robust Speaker Verification

Youngmoon Jung, Joon-Young Yang, Ju-ho Kim, Jaeyoung Roh, Chang Woo Han, Hoon-Young Cho

TL;DR

The paper addresses short-duration speaker verification by rethinking embedding learning rather than solely enhancing encoders. It introduces Duration-Aware Matryoshka Embedding (DAME), which builds a nested hierarchy of prefix embeddings supervised by duration-aligned losses, enabling compact representations for short utterances and richer details for longer ones. DAME is model-agnostic and supports both training from scratch and fine-tuning, showing consistent improvements in short-duration EER on VoxCeleb1-O/E/H and VOiCES without increasing inference cost, across multiple backbones. The work demonstrates robust, duration-aware SV improvements and suggests future directions in adaptive test-time prefix selection and joint duration–margin scheduling.

Abstract

Short-utterance speaker verification remains challenging due to limited speaker-discriminative cues in short speech segments. While existing methods focus on enhancing speaker encoders, the embedding learning strategy still forces a single fixed-dimensional representation reused for utterances of any length, leaving capacity misaligned with the information available at different durations. We propose Duration-Aware Matryoshka Embedding (DAME), a model-agnostic framework that builds a nested hierarchy of sub-embeddings aligned to utterance durations: lower-dimensional representations capture compact speaker traits from short utterances, while higher dimensions encode richer details from longer speech. DAME supports both training from scratch and fine-tuning, and serves as a direct alternative to conventional large-margin fine-tuning, consistently improving performance across durations. On the VoxCeleb1-O/E/H and VOiCES evaluation sets, DAME consistently reduces the equal error rate on 1-s and other short-duration trials, while maintaining full-length performance with no additional inference cost. These gains generalize across various speaker encoder architectures under both general training and fine-tuning setups.

DAME: Duration-Aware Matryoshka Embedding for Duration-Robust Speaker Verification

TL;DR

The paper addresses short-duration speaker verification by rethinking embedding learning rather than solely enhancing encoders. It introduces Duration-Aware Matryoshka Embedding (DAME), which builds a nested hierarchy of prefix embeddings supervised by duration-aligned losses, enabling compact representations for short utterances and richer details for longer ones. DAME is model-agnostic and supports both training from scratch and fine-tuning, showing consistent improvements in short-duration EER on VoxCeleb1-O/E/H and VOiCES without increasing inference cost, across multiple backbones. The work demonstrates robust, duration-aware SV improvements and suggests future directions in adaptive test-time prefix selection and joint duration–margin scheduling.

Abstract

Short-utterance speaker verification remains challenging due to limited speaker-discriminative cues in short speech segments. While existing methods focus on enhancing speaker encoders, the embedding learning strategy still forces a single fixed-dimensional representation reused for utterances of any length, leaving capacity misaligned with the information available at different durations. We propose Duration-Aware Matryoshka Embedding (DAME), a model-agnostic framework that builds a nested hierarchy of sub-embeddings aligned to utterance durations: lower-dimensional representations capture compact speaker traits from short utterances, while higher dimensions encode richer details from longer speech. DAME supports both training from scratch and fine-tuning, and serves as a direct alternative to conventional large-margin fine-tuning, consistently improving performance across durations. On the VoxCeleb1-O/E/H and VOiCES evaluation sets, DAME consistently reduces the equal error rate on 1-s and other short-duration trials, while maintaining full-length performance with no additional inference cost. These gains generalize across various speaker encoder architectures under both general training and fine-tuning setups.
Paper Structure (14 sections, 4 equations, 2 figures, 4 tables)

This paper contains 14 sections, 4 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed method
  4. Overview and Notation
  5. DAME Loss
  6. Training Regime
  7. Experimental Setup
  8. Model Training
  9. Evaluation Protocols
  10. Results
  11. Results on VoxCeleb1
  12. Prefix analysis
  13. Domain robustness on VOiCES
  14. Conclusions

Figures (2)

  • Figure 1: DAME overview (HW, $J{=}K{=}3$): one-to-one duration–prefix alignment; each chunk supervises its matched sub-embedding via a large-margin loss.
  • Figure 2: Relative EER reduction (%) vs. ECAPA-TDNN baseline ($D{=}192$) on VoxCeleb1-O for one DAME-GT-SW model at prefixes $d\in\{48,96,192\}$; results are shown for $\text{f--f}$, $\text{s--avg}$, and 5s--1s.