Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WIND: Accelerated RNN-T Decoding with Windowed Inference for Non-blank Detection

Hainan Xu, Vladimir Bataev, Lilit Grigoryan, Boris Ginsburg

TL;DR

The paper tackles the bottleneck of frame-by-frame RNN-T decoding by introducing WIND, a Windowed Inference strategy that processes a window of frames in parallel to rapidly locate the next non-blank token without sacrificing accuracy. It presents three WIND variants: greedy WIND, batched WIND with label-looping, and a novel WIND beam-search, each designed to reduce decoding latency while preserving or improving accuracy. Empirical results across multiple models and datasets show up to 2.4X speedups in greedy mode and strong speed-accuracy advantages for WIND beam-search compared to existing methods such as ALSD and MAES, with negligible memory overhead and compatibility with CUDA-graph optimizations. The work offers a practical, open-source approach to deploying faster RNN-T inference in diverse ASR tasks, enabling more efficient real-time or low-latency applications.

Abstract

We propose Windowed Inference for Non-blank Detection (WIND), a novel strategy that significantly accelerates RNN-T inference without compromising model accuracy. During model inference, instead of processing frames sequentially, WIND processes multiple frames simultaneously within a window in parallel, allowing the model to quickly locate non-blank predictions during decoding, resulting in significant speed-ups. We implement WIND for greedy decoding, batched greedy decoding with label-looping techniques, and also propose a novel beam-search decoding method. Experiments on multiple datasets with different conditions show that our method, when operating in greedy modes, speeds up as much as 2.4X compared to the baseline sequential approach while maintaining identical Word Error Rate (WER) performance. Our beam-search algorithm achieves slightly better accuracy than alternative methods, with significantly improved speed. We will open-source our WIND implementation.

WIND: Accelerated RNN-T Decoding with Windowed Inference for Non-blank Detection

TL;DR

The paper tackles the bottleneck of frame-by-frame RNN-T decoding by introducing WIND, a Windowed Inference strategy that processes a window of frames in parallel to rapidly locate the next non-blank token without sacrificing accuracy. It presents three WIND variants: greedy WIND, batched WIND with label-looping, and a novel WIND beam-search, each designed to reduce decoding latency while preserving or improving accuracy. Empirical results across multiple models and datasets show up to 2.4X speedups in greedy mode and strong speed-accuracy advantages for WIND beam-search compared to existing methods such as ALSD and MAES, with negligible memory overhead and compatibility with CUDA-graph optimizations. The work offers a practical, open-source approach to deploying faster RNN-T inference in diverse ASR tasks, enabling more efficient real-time or low-latency applications.

Abstract

We propose Windowed Inference for Non-blank Detection (WIND), a novel strategy that significantly accelerates RNN-T inference without compromising model accuracy. During model inference, instead of processing frames sequentially, WIND processes multiple frames simultaneously within a window in parallel, allowing the model to quickly locate non-blank predictions during decoding, resulting in significant speed-ups. We implement WIND for greedy decoding, batched greedy decoding with label-looping techniques, and also propose a novel beam-search decoding method. Experiments on multiple datasets with different conditions show that our method, when operating in greedy modes, speeds up as much as 2.4X compared to the baseline sequential approach while maintaining identical Word Error Rate (WER) performance. Our beam-search algorithm achieves slightly better accuracy than alternative methods, with significantly improved speed. We will open-source our WIND implementation.

Paper Structure

This paper contains 15 sections, 1 equation, 1 figure, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Background: RNN-T
  5. Greedy Inference with WIND Models
  6. A Novel Beam-search Algorithm for WIND
  7. Experiments
  8. Greedy Decoding
  9. Batched Greedy Results
  10. Beam Search Results
  11. Analysis
  12. Memory Footprint
  13. Jump Distribution
  14. Incorporation wth CUDA-graphs
  15. Conclusion and Future Work

Figures (1)

  • Figure 1: Jump distributions of WIND inference with Parakeet-RNNT-1.1b model on Librispeech test-other. Bars are ordered so that smaller jumps are on the left, and the larger right, so readers viewing this on black-and-white printed paper can infer which bar represents which jump interval without using color information.