Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ASTRA: Aligning Speech and Text Representations for Asr without Sampling

Neeraj Gaur, Rohan Agrawal, Gary Wang, Parisa Haghani, Andrew Rosenberg, Bhuvana Ramabhadran

TL;DR

ASTRA addresses the challenge of aligning speech and text representations for end-to-end ASR without sampling-based length matching. It introduces a cross-modal consistency objective $L_c$ that is reinterpreted as a weighted RNNT loss $\\widehat{L_c}$, leveraging alignments learned by RNNT to align speech embeddings and text embeddings without explicit duration modeling. The approach uses unpaired speech/text data with BEST-RQ and RNNT objectives and combines RNNT loss with $L_c$ on paired data. On FLEURS, ASTRA matches or surpasses duration-based text-injection baselines while avoiding duration modeling, and ablations highlight the benefits of MAE consistency and SpecAugment. The work enables pure-text pretraining to benefit ASR and suggests easy LM integration and contextual biasing in multimodal ASR.

Abstract

This paper introduces ASTRA, a novel method for improving Automatic Speech Recognition (ASR) through text injection.Unlike prevailing techniques, ASTRA eliminates the need for sampling to match sequence lengths between speech and text modalities. Instead, it leverages the inherent alignments learned within CTC/RNNT models. This approach offers the following two advantages, namely, avoiding potential misalignment between speech and text features that could arise from upsampling and eliminating the need for models to accurately predict duration of sub-word tokens. This novel formulation of modality (length) matching as a weighted RNNT objective matches the performance of the state-of-the-art duration-based methods on the FLEURS benchmark, while opening up other avenues of research in speech processing.

ASTRA: Aligning Speech and Text Representations for Asr without Sampling

TL;DR

ASTRA addresses the challenge of aligning speech and text representations for end-to-end ASR without sampling-based length matching. It introduces a cross-modal consistency objective that is reinterpreted as a weighted RNNT loss , leveraging alignments learned by RNNT to align speech embeddings and text embeddings without explicit duration modeling. The approach uses unpaired speech/text data with BEST-RQ and RNNT objectives and combines RNNT loss with on paired data. On FLEURS, ASTRA matches or surpasses duration-based text-injection baselines while avoiding duration modeling, and ablations highlight the benefits of MAE consistency and SpecAugment. The work enables pure-text pretraining to benefit ASR and suggests easy LM integration and contextual biasing in multimodal ASR.

Abstract

This paper introduces ASTRA, a novel method for improving Automatic Speech Recognition (ASR) through text injection.Unlike prevailing techniques, ASTRA eliminates the need for sampling to match sequence lengths between speech and text modalities. Instead, it leverages the inherent alignments learned within CTC/RNNT models. This approach offers the following two advantages, namely, avoiding potential misalignment between speech and text features that could arise from upsampling and eliminating the need for models to accurately predict duration of sub-word tokens. This novel formulation of modality (length) matching as a weighted RNNT objective matches the performance of the state-of-the-art duration-based methods on the FLEURS benchmark, while opening up other avenues of research in speech processing.
Paper Structure (12 sections, 10 equations, 3 figures, 3 tables)

This paper contains 12 sections, 10 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. ASTRA model architecture
  4. Model architecture and losses
  5. Learned alignment speech-text consistency
  6. Experimental setup
  7. Architecture
  8. Pretraining
  9. Training and evaluation
  10. Baselines
  11. Results
  12. Conclusion

Figures (3)

  • Figure 1: Model architecture with losses. Light green represents all parts of the model and that are active for unpaired speech data. Similarly, light blue and purple show parts active for paired speech-text data and unpaired text data respectively.
  • Figure 2: Toy example of a speech alignment which consists of non-blank frames, shown in blue, and blank frames, shown in red. As shown consistency is only enforced between speech frames corresponding to non-blank tokens and corresponding text embedding
  • Figure 3: RNNT lattice graves2012sequence where vertical transitions represent non-blank emissions and horizontal transitions represent blank emissions. Also shown is the path for one alignment through the lattice. Here green arrows represent weighted non-blank transitions and red arrows represent weighted blank transitions.