Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Speech Model Pre-training for End-to-End Spoken Language Understanding

Loren Lugosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, Yoshua Bengio

TL;DR

The paper tackles the data-hungry nature of end-to-end spoken language understanding by introducing a pre-training regime that predicts both phonemes and words before fine-tuning on SLU tasks. It presents Fluent Speech Commands, a new open dataset for realistic SLU evaluation, and demonstrates that ASR-target pre-training improves performance on both full and limited-data scenarios, with strong generalization to unseen wordings. The approach leverages a modular architecture (phoneme, word, and intent modules) and a gradual unfreezing schedule to preserve useful pre-trained representations. Overall, the work offers a practical pathway to data-efficient end-to-end SLU and provides an open benchmark for future research on language variability and generalization.

Abstract

Whereas conventional spoken language understanding (SLU) systems map speech to text, and then text to intent, end-to-end SLU systems map speech directly to intent through a single trainable model. Achieving high accuracy with these end-to-end models without a large amount of training data is difficult. We propose a method to reduce the data requirements of end-to-end SLU in which the model is first pre-trained to predict words and phonemes, thus learning good features for SLU. We introduce a new SLU dataset, Fluent Speech Commands, and show that our method improves performance both when the full dataset is used for training and when only a small subset is used. We also describe preliminary experiments to gauge the model's ability to generalize to new phrases not heard during training.

Speech Model Pre-training for End-to-End Spoken Language Understanding

TL;DR

The paper tackles the data-hungry nature of end-to-end spoken language understanding by introducing a pre-training regime that predicts both phonemes and words before fine-tuning on SLU tasks. It presents Fluent Speech Commands, a new open dataset for realistic SLU evaluation, and demonstrates that ASR-target pre-training improves performance on both full and limited-data scenarios, with strong generalization to unseen wordings. The approach leverages a modular architecture (phoneme, word, and intent modules) and a gradual unfreezing schedule to preserve useful pre-trained representations. Overall, the work offers a practical pathway to data-efficient end-to-end SLU and provides an open benchmark for future research on language variability and generalization.

Abstract

Whereas conventional spoken language understanding (SLU) systems map speech to text, and then text to intent, end-to-end SLU systems map speech directly to intent through a single trainable model. Achieving high accuracy with these end-to-end models without a large amount of training data is difficult. We propose a method to reduce the data requirements of end-to-end SLU in which the model is first pre-trained to predict words and phonemes, thus learning good features for SLU. We introduce a new SLU dataset, Fluent Speech Commands, and show that our method improves performance both when the full dataset is used for training and when only a small subset is used. We also describe preliminary experiments to gauge the model's ability to generalize to new phrases not heard during training.

Paper Structure

This paper contains 19 sections, 2 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Dataset
  4. Audio and labels
  5. Data collection
  6. Dataset splits
  7. Related datasets
  8. Model and Pre-training Strategy
  9. Which ASR targets to use?
  10. Phoneme module
  11. Word module
  12. Intent module
  13. Unfreezing schedule
  14. Experiments
  15. Full dataset
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Conventional ASR $\rightarrow$ NLU system for SLU versus end-to-end SLU.
  • Figure 2: The lower layers of the model are pre-trained using ASR targets (words and phonemes). The word and phoneme classifiers are discarded, and the features from the pre-trained part of the model (blue) are used as the input to the subsequent module (white), which is trained using SLU targets.
  • Figure 3: Accuracy on the validation set over time for models trained on (a) the full SLU dataset or (b) 10% of the dataset.