Unsupervised Pre-training of Bidirectional Speech Encoders via Masked Reconstruction

TL;DR

This paper introduces a BERT-style masked reconstruction pre-training objective for bidirectional speech encoders, operating on continuous spectrogram inputs with joint time-frequency masking. By pre-training on large unlabeled corpora (LibriSpeech) and optionally adapting domain information via a linear input network, the approach initializes bidirectional LSTM encoders that can be fine-tuned on smaller labeled datasets (WSJ) for CTC-based ASR. Results across phone- and character-based systems on WSJ and LibriSpeech show that masking width, substantial unlabeled data, and domain adaptation yield consistent, additive gains, with pre-training reducing overfitting and accelerating convergence. The work highlights the viability of unsupervised pre-training for bidirectional speech models and points to future comparisons with other representation-learning methods and hyperparameter selection strategies.

Abstract

We propose an approach for pre-training speech representations via a masked reconstruction loss. Our pre-trained encoder networks are bidirectional and can therefore be used directly in typical bidirectional speech recognition models. The pre-trained networks can then be fine-tuned on a smaller amount of supervised data for speech recognition. Experiments with this approach on the LibriSpeech and Wall Street Journal corpora show promising results. We find that the main factors that lead to speech recognition improvements are: masking segments of sufficient width in both time and frequency, pre-training on a much larger amount of unlabeled data than the labeled data, and domain adaptation when the unlabeled and labeled data come from different domains. The gain from pre-training is additive to that of supervised data augmentation.

Figures (2)

• Figure 1: Illustration of our masked reconstruction approach.
• Figure 2: Dev set learning curves (%CER and CTC loss) of different systems pre-trained on LibriSpeech. The first 5 epochs of fine-tuning update only the LIN and softmax layers.