Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decoder-only Architecture for Speech Recognition with CTC Prompts and Text Data Augmentation

Emiru Tsunoo, Hayato Futami, Yosuke Kashiwagi, Siddhant Arora, Shinji Watanabe

TL;DR

The paper tackles the data scarcity problem in end-to-end ASR by introducing a decoder-only architecture that leverages text-only data through LM-style training and CTC-based audio prompts. Audio information is provided to the decoder via CTC-predicted prompts that compress encoder features, while swapped position encoding helps distinguish prompts from linguistic tokens. A multi-task training scheme jointly optimizes ASR and LM objectives, including a pseudo-prompt LM variant to bridge the gap between speech and text data. Empirical results on LibriSpeech and Switchboard show consistent WER reductions and faster inference compared with traditional encoder–decoder baselines, demonstrating that decoder-only models with text augmentation can effectively exploit large text corpora for ASR. The work suggests that scaling the decoder and extending streaming capabilities could further enhance performance and efficiency in practical ASR systems.

Abstract

Collecting audio-text pairs is expensive; however, it is much easier to access text-only data. Unless using shallow fusion, end-to-end automatic speech recognition (ASR) models require architecture modifications or additional training schemes to use text-only data. Inspired by recent advances in decoder-only language models (LMs), such as GPT-3 and PaLM adopted for speech-processing tasks, we propose using a decoder-only architecture for ASR with simple text augmentation. To provide audio information, encoder features compressed by CTC prediction are used as prompts for the decoder, which can be regarded as refining CTC prediction using the decoder-only model. Because the decoder architecture is the same as an autoregressive LM, it is simple to enhance the model by leveraging external text data with LM training. An experimental comparison using LibriSpeech and Switchboard shows that our proposed models with text augmentation training reduced word error rates from ordinary CTC by 0.3% and 1.4% on LibriSpeech test-clean and testother set, respectively, and 2.9% and 5.0% on Switchboard and CallHome. The proposed model had advantage on computational efficiency compared with conventional encoder-decoder ASR models with a similar parameter setup, and outperformed them on the LibriSpeech 100h and Switchboard training scenarios.

Decoder-only Architecture for Speech Recognition with CTC Prompts and Text Data Augmentation

TL;DR

The paper tackles the data scarcity problem in end-to-end ASR by introducing a decoder-only architecture that leverages text-only data through LM-style training and CTC-based audio prompts. Audio information is provided to the decoder via CTC-predicted prompts that compress encoder features, while swapped position encoding helps distinguish prompts from linguistic tokens. A multi-task training scheme jointly optimizes ASR and LM objectives, including a pseudo-prompt LM variant to bridge the gap between speech and text data. Empirical results on LibriSpeech and Switchboard show consistent WER reductions and faster inference compared with traditional encoder–decoder baselines, demonstrating that decoder-only models with text augmentation can effectively exploit large text corpora for ASR. The work suggests that scaling the decoder and extending streaming capabilities could further enhance performance and efficiency in practical ASR systems.

Abstract

Collecting audio-text pairs is expensive; however, it is much easier to access text-only data. Unless using shallow fusion, end-to-end automatic speech recognition (ASR) models require architecture modifications or additional training schemes to use text-only data. Inspired by recent advances in decoder-only language models (LMs), such as GPT-3 and PaLM adopted for speech-processing tasks, we propose using a decoder-only architecture for ASR with simple text augmentation. To provide audio information, encoder features compressed by CTC prediction are used as prompts for the decoder, which can be regarded as refining CTC prediction using the decoder-only model. Because the decoder architecture is the same as an autoregressive LM, it is simple to enhance the model by leveraging external text data with LM training. An experimental comparison using LibriSpeech and Switchboard shows that our proposed models with text augmentation training reduced word error rates from ordinary CTC by 0.3% and 1.4% on LibriSpeech test-clean and testother set, respectively, and 2.9% and 5.0% on Switchboard and CallHome. The proposed model had advantage on computational efficiency compared with conventional encoder-decoder ASR models with a similar parameter setup, and outperformed them on the LibriSpeech 100h and Switchboard training scenarios.
Paper Structure (13 sections, 7 equations, 2 figures, 4 tables)

This paper contains 13 sections, 7 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. CTC prompts for decoder-only ASR
  3. Decoder-only architecture
  4. Audio prompt using CTC prediction
  5. Training with text augmentation
  6. ASR training
  7. LM training
  8. Experiments
  9. Experimental setup
  10. LibriSpeech 100h experiments
  11. LibriSpeech 960h experiments
  12. Switchboard and Fisher experiments
  13. Conclusion

Figures (2)

  • Figure 1: Model architecture of decoder-only model for ASR with CTC prompts.
  • Figure 2: LM training methods. (a) is ordinary LM training. (b) is a pseudo prompt with the ground truth embedding vectors.