Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Chunk-wise Attention Transducers for Fast and Accurate Streaming Speech-to-Text

Hainan Xu, Vladimir Bataev, Travis M. Bartley, Jagadeesh Balam

TL;DR

The proposed Chunk-wise Attention Transducer (CHAT), a novel extension to RNN-T models that processes audio in fixed-size chunks while employing cross-attention within each chunk, offers a practical solution for deploying more capable streaming speech models without sacrificing real-time constraints.

Abstract

We propose Chunk-wise Attention Transducer (CHAT), a novel extension to RNN-T models that processes audio in fixed-size chunks while employing cross-attention within each chunk. This hybrid approach maintains RNN-T's streaming capability while introducing controlled flexibility for local alignment modeling. CHAT significantly reduces the temporal dimension that RNN-T must handle, yielding substantial efficiency improvements: up to 46.2% reduction in peak training memory, up to 1.36X faster training, and up to 1.69X faster inference. Alongside these efficiency gains, CHAT achieves consistent accuracy improvements over RNN-T across multiple languages and tasks -- up to 6.3% relative WER reduction for speech recognition and up to 18.0% BLEU improvement for speech translation. The method proves particularly effective for speech translation, where RNN-T's strict monotonic alignment hurts performance. Our results demonstrate that the CHAT model offers a practical solution for deploying more capable streaming speech models without sacrificing real-time constraints.

Chunk-wise Attention Transducers for Fast and Accurate Streaming Speech-to-Text

TL;DR

The proposed Chunk-wise Attention Transducer (CHAT), a novel extension to RNN-T models that processes audio in fixed-size chunks while employing cross-attention within each chunk, offers a practical solution for deploying more capable streaming speech models without sacrificing real-time constraints.

Abstract

We propose Chunk-wise Attention Transducer (CHAT), a novel extension to RNN-T models that processes audio in fixed-size chunks while employing cross-attention within each chunk. This hybrid approach maintains RNN-T's streaming capability while introducing controlled flexibility for local alignment modeling. CHAT significantly reduces the temporal dimension that RNN-T must handle, yielding substantial efficiency improvements: up to 46.2% reduction in peak training memory, up to 1.36X faster training, and up to 1.69X faster inference. Alongside these efficiency gains, CHAT achieves consistent accuracy improvements over RNN-T across multiple languages and tasks -- up to 6.3% relative WER reduction for speech recognition and up to 18.0% BLEU improvement for speech translation. The method proves particularly effective for speech translation, where RNN-T's strict monotonic alignment hurts performance. Our results demonstrate that the CHAT model offers a practical solution for deploying more capable streaming speech models without sacrificing real-time constraints.
Paper Structure (18 sections, 7 equations, 2 figures, 5 tables)

This paper contains 18 sections, 7 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. RNN-Transducer
  4. Chunk-based Streaming Encoder
  5. RNN-T Joiner
  6. Method
  7. The Big Picture
  8. Attention Joiner Architecture
  9. Experiments
  10. Speech Recognition Performance
  11. Speech Translation Performance
  12. Computational Efficiency
  13. Analysis
  14. Alignment Visualization
  15. Other Chunk Sizes
  16. ...and 3 more sections

Figures (2)

  • Figure 1: GPU Memory Usage (MB) when training RNNT and CHAT models for one mini-epoch with batch=32 (5000 selected utterances in Librispeech train) on A6000 GPU.
  • Figure 2: From top to bottom: 1. RNN-T frame alignments; 2. CHAT chunk-based alignments; 3. CHAT frame-based alignments. Chunk-size = 12.