Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Problem of Text-To-Speech Model Selection for Synthetic Data Generation in Automatic Speech Recognition

Nick Rossenbach, Ralf Schlüter, Sakriani Sakti

TL;DR

The paper tackles the challenge of selecting TTS models for generating synthetic data to train ASR, focusing on five decoder architectures (Transformer, AR-LSTM, NAR-LSTM, Glow-TTS, Grad-TTS) within a common TTS backbone. It trains a CTC-based ASR from synthetic data and evaluates using WER on LibriSpeech, plus MOS (via NISQA) and synthetic WER, across three data-generation scenarios to assess generalization. The key findings show that AR-LSTM-based TTS provides the best synthetic data for ASR, and that MOS/sWER metrics do not reliably predict downstream ASR performance; a generalization assessment is proposed to gauge TTS extrapolation to new conditions. The work highlights that current perceptual metrics are insufficient for predicting ASR utility of synthetic data and suggests avenues for improved metrics, larger models, and enhanced prosody/duration modeling to close the gap to real data.

Abstract

The rapid development of neural text-to-speech (TTS) systems enabled its usage in other areas of natural language processing such as automatic speech recognition (ASR) or spoken language translation (SLT). Due to the large number of different TTS architectures and their extensions, selecting which TTS systems to use for synthetic data creation is not an easy task. We use the comparison of five different TTS decoder architectures in the scope of synthetic data generation to show the impact on CTC-based speech recognition training. We compare the recognition results to computable metrics like NISQA MOS and intelligibility, finding that there are no clear relations to the ASR performance. We also observe that for data generation auto-regressive decoding performs better than non-autoregressive decoding, and propose an approach to quantify TTS generalization capabilities.

On the Problem of Text-To-Speech Model Selection for Synthetic Data Generation in Automatic Speech Recognition

TL;DR

The paper tackles the challenge of selecting TTS models for generating synthetic data to train ASR, focusing on five decoder architectures (Transformer, AR-LSTM, NAR-LSTM, Glow-TTS, Grad-TTS) within a common TTS backbone. It trains a CTC-based ASR from synthetic data and evaluates using WER on LibriSpeech, plus MOS (via NISQA) and synthetic WER, across three data-generation scenarios to assess generalization. The key findings show that AR-LSTM-based TTS provides the best synthetic data for ASR, and that MOS/sWER metrics do not reliably predict downstream ASR performance; a generalization assessment is proposed to gauge TTS extrapolation to new conditions. The work highlights that current perceptual metrics are insufficient for predicting ASR utility of synthetic data and suggests avenues for improved metrics, larger models, and enhanced prosody/duration modeling to close the gap to real data.

Abstract

The rapid development of neural text-to-speech (TTS) systems enabled its usage in other areas of natural language processing such as automatic speech recognition (ASR) or spoken language translation (SLT). Due to the large number of different TTS architectures and their extensions, selecting which TTS systems to use for synthetic data creation is not an easy task. We use the comparison of five different TTS decoder architectures in the scope of synthetic data generation to show the impact on CTC-based speech recognition training. We compare the recognition results to computable metrics like NISQA MOS and intelligibility, finding that there are no clear relations to the ASR performance. We also observe that for data generation auto-regressive decoding performs better than non-autoregressive decoding, and propose an approach to quantify TTS generalization capabilities.
Paper Structure (17 sections, 4 equations, 2 figures, 3 tables)

This paper contains 17 sections, 4 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Text-to-speech systems
  3. General architecture
  4. Decoder variants
  5. Evaluation pipeline and metrics
  6. Experiments
  7. Data
  8. Automatic speech recogition
  9. Training and model settings
  10. TTS parameter settings
  11. Results
  12. Synthetic data, automatic MOS and recognizability
  13. Decoder characteristics
  14. Generalization
  15. Discussion
  16. ...and 2 more sections

Figures (2)

  • Figure 1: The TTS architecture. We change the decoder marked in red for each of the different systems.
  • Figure 2: Example spectrograms of a selected sequence from the cross validation set. Used TTS decoder from top to bottom: a) Transformer b) AR-LSTM c) Glow-TTS d) Grad-TTS