Improving Accented Speech Recognition using Data Augmentation based on Unsupervised Text-to-Speech Synthesis

TL;DR

The paper addresses the challenge of accented speech recognition with limited labeled data by proposing unsupervised TTS-based data augmentation. It trains TTS on a small accented dataset using pseudo-labels from a baseline ASR, generates substantial synthetic accented speech via independent text prompts, and augments ASR training with both synthetic and non-accented data using a Wav2vec2.0 SSL model. Empirical results show up to 6.1% relative WER reductions on EdAcc in unsupervised setups and up to 7.3% in supervised setups, with gains near the supervised scenario and no detriment to Librispeech performance. The findings demonstrate the practical viability of unsupervised accented data and TTS-based augmentation for improving accented ASR, especially when the evaluation speakers’ first languages overlap with the accented data.

Abstract

This paper investigates the use of unsupervised text-to-speech synthesis (TTS) as a data augmentation method to improve accented speech recognition. TTS systems are trained with a small amount of accented speech training data and their pseudo-labels rather than manual transcriptions, and hence unsupervised. This approach enables the use of accented speech data without manual transcriptions to perform data augmentation for accented speech recognition. Synthetic accented speech data, generated from text prompts by using the TTS systems, are then combined with available non-accented speech data to train automatic speech recognition (ASR) systems. ASR experiments are performed in a self-supervised learning framework using a Wav2vec2.0 model which was pre-trained on large amount of unsupervised accented speech data. The accented speech data for training the unsupervised TTS are read speech, selected from L2-ARCTIC and British Isles corpora, while spontaneous conversational speech from the Edinburgh international accents of English corpus are used as the evaluation data. Experimental results show that Wav2vec2.0 models which are fine-tuned to downstream ASR task with synthetic accented speech data, generated by the unsupervised TTS, yield up to 6.1% relative word error rate reductions compared to a Wav2vec2.0 baseline which is fine-tuned with the non-accented speech data from Librispeech corpus.

Figures (4)

• Figure 1: Unsupervised accented speech training data and theirs pseudo-labels are used to train unsupervised TTS. The pseudo-labels are generated by decoding the unsupervised accented speech training data using the baseline ASR model obtained from the supervised fine-tuning of the SSL pre-trained model with the supervised non-accented speech data. The unsupervised accented speech training data may be included in the semi-supervised fine-tuning for ASR, and the non-accented speech data may be used to train a TTS system.
• Figure 2: Supervised accented speech training data are used to train supervised TTS. These data may be included in the supervised fine-tuning for ASR, and the non-accented speech training data may be used to train a TTS system. The fine-tuned ASR model is used in the ASR inference, with an external language model (LM), to decode test speech.
• Figure 3: Training of VITS parallel end-to-end TTS system. The input text can be either manual transcriptions or pseudo-labels.
• Figure 4: TTS inference using VITS model where text prompt and speaker ID are used as input.