Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ChunkFormer: Masked Chunking Conformer For Long-Form Speech Transcription

Khanh Le, Tuan Vu Ho, Dung Tran, Duc Thanh Chau

TL;DR

ChunkFormer addresses the memory and padding challenges of long-form ASR by introducing chunk-based processing with relative right context, endless decoding, overlapping chunk transformation (OCT), and masked batch decoding. This combination enables transcription of very long audio on low-memory GPUs, achieving up to 980 minutes on a single 80GB GPU and a 7.7% absolute WER improvement on earnings-focused datasets, while preserving accuracy on shorter tasks. The architecture integrates a convolutional enhanced chunking scheme with limited-context attention to dramatically reduce memory and compute costs, delivering up to a 3.7x efficiency gain over prior state-of-the-art models. These results demonstrate the practicality of ChunkFormer for industrial-scale ASR deployments, offering scalable, cost-effective long-form transcription without sacrificing short-form performance.

Abstract

Deploying ASR models at an industrial scale poses significant challenges in hardware resource management, especially for long-form transcription tasks where audio may last for hours. Large Conformer models, despite their capabilities, are limited to processing only 15 minutes of audio on an 80GB GPU. Furthermore, variable input lengths worsen inefficiencies, as standard batching leads to excessive padding, increasing resource consumption and execution time. To address this, we introduce ChunkFormer, an efficient ASR model that uses chunk-wise processing with relative right context, enabling long audio transcriptions on low-memory GPUs. ChunkFormer handles up to 16 hours of audio on an 80GB GPU, 1.5x longer than the current state-of-the-art FastConformer, while also boosting long-form transcription performance with up to 7.7% absolute reduction on word error rate and maintaining accuracy on shorter tasks compared to Conformer. By eliminating the need for padding in standard batching, ChunkFormer's masked batching technique reduces execution time and memory usage by more than 3x in batch processing, substantially reducing costs for a wide range of ASR systems, particularly regarding GPU resources for models serving in real-world applications.

ChunkFormer: Masked Chunking Conformer For Long-Form Speech Transcription

TL;DR

ChunkFormer addresses the memory and padding challenges of long-form ASR by introducing chunk-based processing with relative right context, endless decoding, overlapping chunk transformation (OCT), and masked batch decoding. This combination enables transcription of very long audio on low-memory GPUs, achieving up to 980 minutes on a single 80GB GPU and a 7.7% absolute WER improvement on earnings-focused datasets, while preserving accuracy on shorter tasks. The architecture integrates a convolutional enhanced chunking scheme with limited-context attention to dramatically reduce memory and compute costs, delivering up to a 3.7x efficiency gain over prior state-of-the-art models. These results demonstrate the practicality of ChunkFormer for industrial-scale ASR deployments, offering scalable, cost-effective long-form transcription without sacrificing short-form performance.

Abstract

Deploying ASR models at an industrial scale poses significant challenges in hardware resource management, especially for long-form transcription tasks where audio may last for hours. Large Conformer models, despite their capabilities, are limited to processing only 15 minutes of audio on an 80GB GPU. Furthermore, variable input lengths worsen inefficiencies, as standard batching leads to excessive padding, increasing resource consumption and execution time. To address this, we introduce ChunkFormer, an efficient ASR model that uses chunk-wise processing with relative right context, enabling long audio transcriptions on low-memory GPUs. ChunkFormer handles up to 16 hours of audio on an 80GB GPU, 1.5x longer than the current state-of-the-art FastConformer, while also boosting long-form transcription performance with up to 7.7% absolute reduction on word error rate and maintaining accuracy on shorter tasks compared to Conformer. By eliminating the need for padding in standard batching, ChunkFormer's masked batching technique reduces execution time and memory usage by more than 3x in batch processing, substantially reducing costs for a wide range of ASR systems, particularly regarding GPU resources for models serving in real-world applications.

Paper Structure

This paper contains 9 sections, 7 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Endless Decoding in ChunkFormer
  4. Masked Batch Decoding
  5. ChunkFormer Convolution
  6. Experiments
  7. Experimental Setups
  8. Results
  9. Conclusions

Figures (4)

  • Figure 1: The ChunkFormer model architecture
  • Figure 2: Visualization of receptive field expansion in the $i^{th}$ encoder layer of ChunkFormer with $c = 3$, $r = 2$ and $N=4$.
  • Figure 3: Illustration of Overlapping Chunk Transformation with $l = 4$, $c = 3$, and $r = 2$, assuming the first 15 frames have been decoded. Padding values of -1 are applied to the last chunk for chunk batching and to the flattened sequence for overlapping segmentation. Gray (padding) areas are masked during convolution and attention computation.
  • Figure 4: Masked batch with $l = 4$, $c = 3$, and $r = 2$. (a) shows the output from the OCT, and (b) displays the corresponding mask. Bold-bordered areas represent the current batch of chunks, with $m_x = 5$, $n_x = 8$, and $m_y = 3$. The numbers represent frame indices, and since $\mathbf{X}$ consists of 23 frames, a padding value of -1 is applied to $\mathbf{X}^7$ to complete a chunk. Gray areas are masked during the convolution and attention computations using (b) mask.