Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Speed of Light Exact Greedy Decoding for RNN-T Speech Recognition Models on GPU

Daniel Galvez, Vladimir Bataev, Hainan Xu, Tim Kaldewey

TL;DR

This work tackles the bottleneck of greedy decoding in GPU-based RNN-T speech recognition by introducing an exact GPU-based greedy decoding method that leverages CUDA Graphs with conditional nodes to eliminate GPU idle time. By removing host synchronization and enabling data-dependent control flow entirely on the device, the authors achieve substantial end-to-end speedups (up to 2.51x for a 1.1B Parakeet RNN-T and 2.70x when combined with label looping) and significant decoder-only improvements (up to 6.59x, or 10.40x with label looping). The results demonstrate that a 1.1B RNN-T can approach CTC throughput, challenging the belief that autoregressive models are impractical for high-throughput inference. The method is complementary to existing techniques, generalizes to other autoregressive models, and is available in NVIDIA NeMo.

Abstract

The vast majority of inference time for RNN Transducer (RNN-T) models today is spent on decoding. Current state-of-the-art RNN-T decoding implementations leave the GPU idle ~80% of the time. Leveraging a new CUDA 12.4 feature, CUDA graph conditional nodes, we present an exact GPU-based implementation of greedy decoding for RNN-T models that eliminates this idle time. Our optimizations speed up a 1.1 billion parameter RNN-T model end-to-end by a factor of 2.5x. This technique can applied to the "label looping" alternative greedy decoding algorithm as well, achieving 1.7x and 1.4x end-to-end speedups when applied to 1.1 billion parameter RNN-T and Token and Duration Transducer models respectively. This work enables a 1.1 billion parameter RNN-T model to run only 16% slower than a similarly sized CTC model, contradicting the common belief that RNN-T models are not suitable for high throughput inference. The implementation is available in NVIDIA NeMo.

Speed of Light Exact Greedy Decoding for RNN-T Speech Recognition Models on GPU

TL;DR

This work tackles the bottleneck of greedy decoding in GPU-based RNN-T speech recognition by introducing an exact GPU-based greedy decoding method that leverages CUDA Graphs with conditional nodes to eliminate GPU idle time. By removing host synchronization and enabling data-dependent control flow entirely on the device, the authors achieve substantial end-to-end speedups (up to 2.51x for a 1.1B Parakeet RNN-T and 2.70x when combined with label looping) and significant decoder-only improvements (up to 6.59x, or 10.40x with label looping). The results demonstrate that a 1.1B RNN-T can approach CTC throughput, challenging the belief that autoregressive models are impractical for high-throughput inference. The method is complementary to existing techniques, generalizes to other autoregressive models, and is available in NVIDIA NeMo.

Abstract

The vast majority of inference time for RNN Transducer (RNN-T) models today is spent on decoding. Current state-of-the-art RNN-T decoding implementations leave the GPU idle ~80% of the time. Leveraging a new CUDA 12.4 feature, CUDA graph conditional nodes, we present an exact GPU-based implementation of greedy decoding for RNN-T models that eliminates this idle time. Our optimizations speed up a 1.1 billion parameter RNN-T model end-to-end by a factor of 2.5x. This technique can applied to the "label looping" alternative greedy decoding algorithm as well, achieving 1.7x and 1.4x end-to-end speedups when applied to 1.1 billion parameter RNN-T and Token and Duration Transducer models respectively. This work enables a 1.1 billion parameter RNN-T model to run only 16% slower than a similarly sized CTC model, contradicting the common belief that RNN-T models are not suitable for high throughput inference. The implementation is available in NVIDIA NeMo.
Paper Structure (8 sections, 3 figures, 1 table, 1 algorithm)

This paper contains 8 sections, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Algorithm Implementation
  4. Removal of host synchronization
  5. Inserting data-dependent control flow
  6. Results
  7. Future Work
  8. Conclusion

Figures (3)

  • Figure 1: Time spent in the greedy decoder vs. the encoder in NVIDIA NeMo's Parakeet RNN-T 1.1b model to transcribe Librispeech Test Other, a 5.1 hour dataset, at batch size 32. The total runtime is 44.28 seconds on an A100-80GiB GPU.
  • Figure 2: Nsight Systems profile of one "outer time step" iteration of the loop in line \ref{['alg:outerloop']} of Algorithm \ref{['rnntgreedydecoding']}. Notice that the "CUDA HW" row above has several blank regions. This indicates that the GPU is idle during those times.
  • Figure 3: Schematic diagram of a CUDA graph with a while loop. The left hand side chain is a CUDA graph of nodes, whose B node is a while loop type "conditional node." This node itself contains a "child" CUDA graph within it.