Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Acquiring Pronunciation Knowledge from Transcribed Speech Audio via Multi-task Learning

Siqi Sun, Korin Richmond

TL;DR

The paper addresses pronunciation errors in bootstrapped Seq2Seq frontends for TTS caused by limited lexical coverage. It introduces a multi-task learning framework that adds an acoustic regression task, trained on transcribed speech, to transfer pronunciation knowledge to extra-exclusive word types without requiring ASR training. A novel multi-task model with an Acoustic Decoder is proposed, using a pre-trained frontend to generate pseudo-pronunciations for transcribed-speech data and optimizing a pronunciation loss together with an acoustic loss. Experiments show a substantial reduction in $PER$ for extra-exclusive words (from $2.5\%$ to $1.6\%$), achieving performance on par with an FA-based approach but with a simpler, offline pre-train$\rightarrow$re-train workflow, enabling robust pronunciation generalization for unseen words.

Abstract

Recent work has shown the feasibility and benefit of bootstrapping an integrated sequence-to-sequence (Seq2Seq) linguistic frontend from a traditional pipeline-based frontend for text-to-speech (TTS). To overcome the fixed lexical coverage of bootstrapping training data, previous work has proposed to leverage easily accessible transcribed speech audio as an additional training source for acquiring novel pronunciation knowledge for uncovered words, which relies on an auxiliary ASR model as part of a cumbersome implementation flow. In this work, we propose an alternative method to leverage transcribed speech audio as an additional training source, based on multi-task learning (MTL). Experiments show that, compared to a baseline Seq2Seq frontend, the proposed MTL-based method reduces PER from 2.5% to 1.6% for those word types covered exclusively in transcribed speech audio, achieving a similar performance to the previous method but with a much simpler implementation flow.

Acquiring Pronunciation Knowledge from Transcribed Speech Audio via Multi-task Learning

TL;DR

The paper addresses pronunciation errors in bootstrapped Seq2Seq frontends for TTS caused by limited lexical coverage. It introduces a multi-task learning framework that adds an acoustic regression task, trained on transcribed speech, to transfer pronunciation knowledge to extra-exclusive word types without requiring ASR training. A novel multi-task model with an Acoustic Decoder is proposed, using a pre-trained frontend to generate pseudo-pronunciations for transcribed-speech data and optimizing a pronunciation loss together with an acoustic loss. Experiments show a substantial reduction in for extra-exclusive words (from to ), achieving performance on par with an FA-based approach but with a simpler, offline pre-trainre-train workflow, enabling robust pronunciation generalization for unseen words.

Abstract

Recent work has shown the feasibility and benefit of bootstrapping an integrated sequence-to-sequence (Seq2Seq) linguistic frontend from a traditional pipeline-based frontend for text-to-speech (TTS). To overcome the fixed lexical coverage of bootstrapping training data, previous work has proposed to leverage easily accessible transcribed speech audio as an additional training source for acquiring novel pronunciation knowledge for uncovered words, which relies on an auxiliary ASR model as part of a cumbersome implementation flow. In this work, we propose an alternative method to leverage transcribed speech audio as an additional training source, based on multi-task learning (MTL). Experiments show that, compared to a baseline Seq2Seq frontend, the proposed MTL-based method reduces PER from 2.5% to 1.6% for those word types covered exclusively in transcribed speech audio, achieving a similar performance to the previous method but with a much simpler implementation flow.
Paper Structure (12 sections, 3 equations, 2 figures, 3 tables)

This paper contains 12 sections, 3 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Seq2Seq Frontend
  4. Acquiring Pronunciation from Transcribed Speech Audio
  5. Multi-task Learning in Pronunciation Modelling
  6. Method
  7. Multi-task Model architecture
  8. Experiments
  9. Experimental Setup
  10. Results
  11. Analyses
  12. Conclusion

Figures (2)

  • Figure 1: The two stages involved in our MTL-based method.
  • Figure 2: The multi-task model architecture. The dashed arrows indicate the Acoustic Decoder is integrated by attending to only one of the intermediate representations of the standard Seq2Seq frontend (i.e., Text Encoder + Pronunciation Decoder. See Sec. \ref{['seq2seq']}). GS stands for Gumbel-Softmax.