Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MATE: Matryoshka Audio-Text Embeddings for Open-Vocabulary Keyword Spotting

Youngmoon Jung, Myunghun Jung, Joon-Young Yang, Yong-Hyeok Lee, Jaeyoung Roh, Hoon-Young Cho

TL;DR

This work tackles open-vocabulary keyword spotting with text enrollment by introducing Matryoshka Audio--Text Embeddings (MATE), a dual-encoder that embeds multiple granularities within a single vector via nested prefixes. A corpus-wide PCA-based target guides alignment of acoustic and text prefixes, concentrating salient keyword cues in low-dimensional portions while higher dimensions add detail, all trained with standard deep metric learning objectives. Across WSJ and LibriPhrase benchmarks, MATE achieves state-of-the-art or competitive results, notably improving Average Precision on WSJ and reducing cross-corpus errors on LibriPhrase. The approach preserves inference cost and offers stability through a delayed alignment schedule, with potential extensions to phoneme-level matching and alternative compression techniques like LDA.

Abstract

Open-vocabulary keyword spotting (KWS) with text-based enrollment has emerged as a flexible alternative to fixed-phrase triggers. Prior utterance-level matching methods, from an embedding-learning standpoint, learn embeddings at a single fixed dimensionality. We depart from this design and propose Matryoshka Audio-Text Embeddings (MATE), a dual-encoder framework that encodes multiple embedding granularities within a single vector via nested sub-embeddings ("prefixes"). Specifically, we introduce a PCA-guided prefix alignment: PCA-compressed versions of the full text embedding for each prefix size serve as teacher targets to align both audio and text prefixes. This alignment concentrates salient keyword cues in lower-dimensional prefixes, while higher dimensions add detail. MATE is trained with standard deep metric learning objectives for audio-text KWS, and is loss-agnostic. To our knowledge, this is the first application of matryoshka-style embeddings to KWS, achieving state-of-the-art results on WSJ and LibriPhrase without any inference overhead.

MATE: Matryoshka Audio-Text Embeddings for Open-Vocabulary Keyword Spotting

TL;DR

This work tackles open-vocabulary keyword spotting with text enrollment by introducing Matryoshka Audio--Text Embeddings (MATE), a dual-encoder that embeds multiple granularities within a single vector via nested prefixes. A corpus-wide PCA-based target guides alignment of acoustic and text prefixes, concentrating salient keyword cues in low-dimensional portions while higher dimensions add detail, all trained with standard deep metric learning objectives. Across WSJ and LibriPhrase benchmarks, MATE achieves state-of-the-art or competitive results, notably improving Average Precision on WSJ and reducing cross-corpus errors on LibriPhrase. The approach preserves inference cost and offers stability through a delayed alignment schedule, with potential extensions to phoneme-level matching and alternative compression techniques like LDA.

Abstract

Open-vocabulary keyword spotting (KWS) with text-based enrollment has emerged as a flexible alternative to fixed-phrase triggers. Prior utterance-level matching methods, from an embedding-learning standpoint, learn embeddings at a single fixed dimensionality. We depart from this design and propose Matryoshka Audio-Text Embeddings (MATE), a dual-encoder framework that encodes multiple embedding granularities within a single vector via nested sub-embeddings ("prefixes"). Specifically, we introduce a PCA-guided prefix alignment: PCA-compressed versions of the full text embedding for each prefix size serve as teacher targets to align both audio and text prefixes. This alignment concentrates salient keyword cues in lower-dimensional prefixes, while higher dimensions add detail. MATE is trained with standard deep metric learning objectives for audio-text KWS, and is loss-agnostic. To our knowledge, this is the first application of matryoshka-style embeddings to KWS, achieving state-of-the-art results on WSJ and LibriPhrase without any inference overhead.
Paper Structure (13 sections, 11 equations, 1 figure, 5 tables)

This paper contains 13 sections, 11 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Method
  4. Overview and Notation
  5. Matryoshka Audio--Text Embeddings (MATE)
  6. Experiments
  7. Experimental Setup
  8. Multi-scale supervision under RPL
  9. WSJ main results
  10. Effect of the number of prefixes $K$
  11. Effect of alignment composition
  12. Performance on LibriPhrase
  13. Conclusion

Figures (1)

  • Figure 1: Overview of MATE. A corpus-wide text inner-dependency matrix $\bar{\boldsymbol{A}}^{D}_t$ is estimated and factorized (SVD) to obtain projection heads $\{\boldsymbol{A}^{d_k}_t\}$. For each $d_k$, the acoustic/text prefixes $\boldsymbol{u}^{k}_a,\boldsymbol{u}^{k}_t$ (leading $d_k$-dim sub-vectors) are aligned to the PCA-compressed text sub-embedding $\tilde{\boldsymbol{u}}^{k}_t$; the full-dim pair uses the main loss (RPL).