Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pretraining End-to-End Keyword Search with Automatically Discovered Acoustic Units

Bolaji Yusuf, Jan "Honza" Černocký, Murat Saraçlar

TL;DR

This work tackles the gap where end-to-end keyword search (KWS) lags behind ASR-based approaches by introducing pretraining on untranscribed speech through acoustic unit discovery (AUD). The authors employ Hierarchical Subspace Hidden Markov Models (H-SHMM) to produce discrete acoustic units, form pseudo-words from unit sequences, and pretrain an E2E KWS model before finetuning on a small transcribed corpus; they also transfer the document encoder and switch to a grapheme-based query encoder during fine-tuning. Experiments on English Libri-light and Turkish BNTR show that AUD-based pretraining yields substantial ATWV gains with MFCC inputs, and even larger gains when using XLS-R-derived features for AUD, with gains correlating with AUD quality as measured by NMI. These results demonstrate a practical route to leverage large unlabeled speech for improving E2E KWS, with potential for scalable multilingual pretraining and improved unit-to-word segmentation strategies.

Abstract

End-to-end (E2E) keyword search (KWS) has emerged as an alternative and complimentary approach to conventional keyword search which depends on the output of automatic speech recognition (ASR) systems. While E2E methods greatly simplify the KWS pipeline, they generally have worse performance than their ASR-based counterparts, which can benefit from pretraining with untranscribed data. In this work, we propose a method for pretraining E2E KWS systems with untranscribed data, which involves using acoustic unit discovery (AUD) to obtain discrete units for untranscribed data and then learning to locate sequences of such units in the speech. We conduct experiments across languages and AUD systems: we show that finetuning such a model significantly outperforms a model trained from scratch, and the performance improvements are generally correlated with the quality of the AUD system used for pretraining.

Pretraining End-to-End Keyword Search with Automatically Discovered Acoustic Units

TL;DR

This work tackles the gap where end-to-end keyword search (KWS) lags behind ASR-based approaches by introducing pretraining on untranscribed speech through acoustic unit discovery (AUD). The authors employ Hierarchical Subspace Hidden Markov Models (H-SHMM) to produce discrete acoustic units, form pseudo-words from unit sequences, and pretrain an E2E KWS model before finetuning on a small transcribed corpus; they also transfer the document encoder and switch to a grapheme-based query encoder during fine-tuning. Experiments on English Libri-light and Turkish BNTR show that AUD-based pretraining yields substantial ATWV gains with MFCC inputs, and even larger gains when using XLS-R-derived features for AUD, with gains correlating with AUD quality as measured by NMI. These results demonstrate a practical route to leverage large unlabeled speech for improving E2E KWS, with potential for scalable multilingual pretraining and improved unit-to-word segmentation strategies.

Abstract

End-to-end (E2E) keyword search (KWS) has emerged as an alternative and complimentary approach to conventional keyword search which depends on the output of automatic speech recognition (ASR) systems. While E2E methods greatly simplify the KWS pipeline, they generally have worse performance than their ASR-based counterparts, which can benefit from pretraining with untranscribed data. In this work, we propose a method for pretraining E2E KWS systems with untranscribed data, which involves using acoustic unit discovery (AUD) to obtain discrete units for untranscribed data and then learning to locate sequences of such units in the speech. We conduct experiments across languages and AUD systems: we show that finetuning such a model significantly outperforms a model trained from scratch, and the performance improvements are generally correlated with the quality of the AUD system used for pretraining.
Paper Structure (16 sections, 8 equations, 1 figure, 2 tables)

This paper contains 16 sections, 8 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Background
  4. End-to-end Keyword Search
  5. Hierarchical Subspace Hidden Markov Model
  6. Pretraining KWS with AUD
  7. Experiments
  8. Setup
  9. Datasets
  10. Acoustic features
  11. Metrics
  12. Model configuration and hyper-parameters
  13. Training details
  14. Results
  15. Conclusions
  16. ...and 1 more sections

Figures (1)

  • Figure 1: E2E KWS system during pretraining and finetuning. Grey boxes: non-trainable components, white boxes: components which are trained from scratch, blue boxes: trainable components whose parameters are transferred from a pretrained model.