Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do we really need Self-Attention for Streaming Automatic Speech Recognition?

Youness Dkhissi, Valentin Vielzeuf, Elys Allesiardo, Anthony Larcher

TL;DR

The paper investigates whether Self-Attention is necessary for streaming automatic speech recognition (ASR), where input is processed in fixed-size chunks. Through a strict chunked analysis of a Conformer-Transducer, the authors reveal that Self-Attention mainly captures local dependencies within a chunk, acting as an expensive operator when global context is unavailable. They propose two encoder variants: a hard approach that removes Self-Attention and a soft approach that substitutes it with a 1-D deformable convolution, both preserving accuracy while substantially reducing parameters and Real-Time Factor (RTF). On LibriSpeech and TEDLIUM-2, these variants achieve comparable or better Word Error Rate with notable efficiency gains, suggesting streaming-optimized architectures should favor local-pattern processing over global attention. This work motivates revisiting streaming ASR designs toward convolution-focused, low-latency solutions, potentially inspired by ContextNet and QuartzNet architectures.

Abstract

Transformer-based architectures are the most used architectures in many deep learning fields like Natural Language Processing, Computer Vision or Speech processing. It may encourage the direct use of Transformers in the constrained tasks, without questioning whether it will yield the same benefits as in standard tasks. Given specific constraints, it is essential to evaluate the relevance of transformer models. This work questions the suitability of transformers for specific domains. We argue that the high computational requirements and latency issues associated with these models do not align well with streaming applications. Our study promotes the search for alternative strategies to improve efficiency without sacrificing performance. In light of this observation, our paper critically examines the usefulness of transformer architecture in such constrained environments. As a first attempt, we show that the computational cost for Streaming Automatic Speech Recognition (ASR) can be reduced using deformable convolution instead of Self-Attention. Furthermore, we show that Self-Attention mechanisms can be entirely removed and not replaced, without observing significant degradation in the Word Error Rate.

Do we really need Self-Attention for Streaming Automatic Speech Recognition?

TL;DR

The paper investigates whether Self-Attention is necessary for streaming automatic speech recognition (ASR), where input is processed in fixed-size chunks. Through a strict chunked analysis of a Conformer-Transducer, the authors reveal that Self-Attention mainly captures local dependencies within a chunk, acting as an expensive operator when global context is unavailable. They propose two encoder variants: a hard approach that removes Self-Attention and a soft approach that substitutes it with a 1-D deformable convolution, both preserving accuracy while substantially reducing parameters and Real-Time Factor (RTF). On LibriSpeech and TEDLIUM-2, these variants achieve comparable or better Word Error Rate with notable efficiency gains, suggesting streaming-optimized architectures should favor local-pattern processing over global attention. This work motivates revisiting streaming ASR designs toward convolution-focused, low-latency solutions, potentially inspired by ContextNet and QuartzNet architectures.

Abstract

Transformer-based architectures are the most used architectures in many deep learning fields like Natural Language Processing, Computer Vision or Speech processing. It may encourage the direct use of Transformers in the constrained tasks, without questioning whether it will yield the same benefits as in standard tasks. Given specific constraints, it is essential to evaluate the relevance of transformer models. This work questions the suitability of transformers for specific domains. We argue that the high computational requirements and latency issues associated with these models do not align well with streaming applications. Our study promotes the search for alternative strategies to improve efficiency without sacrificing performance. In light of this observation, our paper critically examines the usefulness of transformer architecture in such constrained environments. As a first attempt, we show that the computational cost for Streaming Automatic Speech Recognition (ASR) can be reduced using deformable convolution instead of Self-Attention. Furthermore, we show that Self-Attention mechanisms can be entirely removed and not replaced, without observing significant degradation in the Word Error Rate.
Paper Structure (12 sections, 1 equation, 3 figures, 4 tables)

This paper contains 12 sections, 1 equation, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. preliminary analysis
  3. Attention replacement
  4. Hard Approach
  5. Soft Approach
  6. Experiments
  7. Evaluation protocol
  8. Model configuration
  9. Results
  10. Ablation studies
  11. Conclusion
  12. Acknowledgements

Figures (3)

  • Figure 1: Illustration of the mean attention map per layer using chunk size 1280 ms on LibriSpeech test-clean dataset.
  • Figure 2: Illustration of a 1-D standard convolution (left) and 1-D deformable convolution (right) with kernel size of 3 applied at the $3^{rd}$ timestep. The coloured squares represent the elements on which the kernel of each convolution will be applied. The offsets applied to the deformable convolution in this example are [-1,3,0].
  • Figure 3: Comparison of Real Time Factor (RTF) between the baseline, soft and hard approach for each utterance on LibriSpeech test-clean dataset. In \ref{['subfig:gpu_aug']}, the utterances have been repeated to be x3 longer.