BEST-STD: Bidirectional Mamba-Enhanced Speech Tokenization for Spoken Term Detection
Anup Singh, Kris Demuynck, Vipul Arora
TL;DR
This work tackles spoken term detection by moving away from frame-level features and DTW-based matching toward discrete, speaker-agnostic speech tokens. It introduces BEST-STD, which uses a bidirectional Mamba encoder to produce contextual frame embeddings that are quantized into tokens via a vector quantizer, trained with a self-supervised, DTW-aligned contrastive objective to ensure consistency across utterances of the same term. An inverted index with bigram tokens enables fast, scalable retrieval using Jaccard similarity, robust to out-of-vocabulary terms. Empirical results on LibriSpeech and TIMIT show that BEST-STD outperforms traditional DTW baselines and existing tokenizers in both accuracy and efficiency, with the bidirectional Mamba providing superior temporal modeling compared to Transformers.
Abstract
Spoken term detection (STD) is often hindered by reliance on frame-level features and the computationally intensive DTW-based template matching, limiting its practicality. To address these challenges, we propose a novel approach that encodes speech into discrete, speaker-agnostic semantic tokens. This facilitates fast retrieval using text-based search algorithms and effectively handles out-of-vocabulary terms. Our approach focuses on generating consistent token sequences across varying utterances of the same term. We also propose a bidirectional state space modeling within the Mamba encoder, trained in a self-supervised learning framework, to learn contextual frame-level features that are further encoded into discrete tokens. Our analysis shows that our speech tokens exhibit greater speaker invariance than those from existing tokenizers, making them more suitable for STD tasks. Empirical evaluation on LibriSpeech and TIMIT databases indicates that our method outperforms existing STD baselines while being more efficient.