Semi-supervised Sequence-to-sequence ASR using Unpaired Speech and Text

TL;DR

<3-5 sentence high-level summary> This paper tackles data scarcity in end-to-end sequence-to-sequence ASR by introducing a semi-supervised, cycle-consistency framework that leverages unpaired speech and text through ASR↔TTS collaborations. It proposes a new end-to-end differentiable ASR→TTS loss augmented with speaker information via x-vectors, complemented by a TTS→ASR path, and jointly trains these losses when both data types are available. Extensive experiments on WSJ and Librispeech show that unpaired text data is particularly valuable in very low-resource regimes, while combining unpaired speech and text yields additional gains; results are competitive with or surpass several prior methods, with further improvements from RNNLM fusion. The work provides practical guidance on data modality choices and demonstrates the effectiveness of end-to-end semi-supervised training for ASR, with code released in ESPnet for reproducibility.

Abstract

Sequence-to-sequence automatic speech recognition (ASR) models require large quantities of data to attain high performance. For this reason, there has been a recent surge in interest for unsupervised and semi-supervised training in such models. This work builds upon recent results showing notable improvements in semi-supervised training using cycle-consistency and related techniques. Such techniques derive training procedures and losses able to leverage unpaired speech and/or text data by combining ASR with Text-to-Speech (TTS) models. In particular, this work proposes a new semi-supervised loss combining an end-to-end differentiable ASR$\rightarrow$TTS loss with TTS$\rightarrow$ASR loss. The method is able to leverage both unpaired speech and text data to outperform recently proposed related techniques in terms of \%WER. We provide extensive results analyzing the impact of data quantity and speech and text modalities and show consistent gains across WSJ and Librispeech corpora. Our code is provided in ESPnet to reproduce the experiments.

Figures (1)

• Figure 1: Simplified representation of the cycle-consistency training for unpaired speech and text. The flow of the diagram is bottom-to-top. The red dotted lines denotes the gradient propagation and the updated parameters. Figure \ref{['fig:overview']}a denotes the ASR$\rightarrow$TTS pipeline where $\mathbf{Y}_{k}$ is the $k^{\mathrm{th}}$ randomly sampled ASR output. Figure \ref{['fig:overview']}b shows the TTS$\rightarrow$ASR pipeline.