Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A low latency attention module for streaming self-supervised speech representation learning

Jianbo Ma, Siqi Pan, Deepak Chandran, Andrea Fanelli, Richard Cartwright

TL;DR

An implementation of the attention module that enables training of SSRL architectures with low compute and memory requirements, while allowing real-time inference with low and fixed latency is presented.

Abstract

The transformer is a fundamental building block in deep learning, and the attention mechanism is the transformer's core component. Self-supervised speech representation learning (SSRL) represents a popular use-case for the transformer architecture. Due to transformers' acausal behavior, the use of transformers for SSRL has been predominantly focused on acausal applications. However, several media processing problems, such as speech processing, require real-time solutions. In this paper, we present an implementation of the attention module that enables training of SSRL architectures with low compute and memory requirements, while allowing real-time inference with low and fixed latency. The attention module proposed in this paper includes two components, streaming attention (SA) and low-latency streaming attention (LLSA). The SA represents our proposal for an efficient streaming SSRL implementation, while the LLSA solves the latency build-up problem of other streaming attention architectures, such as the masked acausal attention (MAA), guaranteeing a latency equal to one layer even when multiple layers are stacked. We present a comparative analysis between the vanilla attention, which we will refer here as acausal attention (AA), the SA, and the LLSA, by training a streaming SSRL with automatic speech recognition as downstream task. When training on librispeech-clean-100 and testing on librispeech-test-clean, our low-latency attention module has a word error rate (WER) of 5.84%, which represents a significant improvement over the MAA (WER = 13.82%). Our implementation also reduces the inference latency from 1.92 to 0.16 seconds. The proposed low-latency module preserves many of the benefits of conventional acausal transformers, but also enables latency characteristics that make it applicable to real-time streaming applications.

A low latency attention module for streaming self-supervised speech representation learning

TL;DR

An implementation of the attention module that enables training of SSRL architectures with low compute and memory requirements, while allowing real-time inference with low and fixed latency is presented.

Abstract

The transformer is a fundamental building block in deep learning, and the attention mechanism is the transformer's core component. Self-supervised speech representation learning (SSRL) represents a popular use-case for the transformer architecture. Due to transformers' acausal behavior, the use of transformers for SSRL has been predominantly focused on acausal applications. However, several media processing problems, such as speech processing, require real-time solutions. In this paper, we present an implementation of the attention module that enables training of SSRL architectures with low compute and memory requirements, while allowing real-time inference with low and fixed latency. The attention module proposed in this paper includes two components, streaming attention (SA) and low-latency streaming attention (LLSA). The SA represents our proposal for an efficient streaming SSRL implementation, while the LLSA solves the latency build-up problem of other streaming attention architectures, such as the masked acausal attention (MAA), guaranteeing a latency equal to one layer even when multiple layers are stacked. We present a comparative analysis between the vanilla attention, which we will refer here as acausal attention (AA), the SA, and the LLSA, by training a streaming SSRL with automatic speech recognition as downstream task. When training on librispeech-clean-100 and testing on librispeech-test-clean, our low-latency attention module has a word error rate (WER) of 5.84%, which represents a significant improvement over the MAA (WER = 13.82%). Our implementation also reduces the inference latency from 1.92 to 0.16 seconds. The proposed low-latency module preserves many of the benefits of conventional acausal transformers, but also enables latency characteristics that make it applicable to real-time streaming applications.
Paper Structure (24 sections, 16 equations, 6 figures, 3 tables)

This paper contains 24 sections, 16 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Acausal Attention
  4. Masked Acausal Attention
  5. HuBERT
  6. Low latency attention module
  7. Streaming Attention
  8. Forward Propagation
  9. Back-propagation
  10. Derivative with respect to values
  11. Derivative with respect to queries
  12. Derivative with respect to keys
  13. Low Latency Streaming Attention
  14. SHuBERT: HuBERT with low latency attention module
  15. Experimental setup
  16. ...and 9 more sections

Figures (6)

  • Figure 1: Scaled Dot-Product Attention unit.
  • Figure 2: HuBERT Model Architecture hsu2021hubert
  • Figure 3: SHuBERT Model Architecture
  • Figure 4: Illustration of latency for AA, SA and LLSA. The horizontal axis represents time slice and $t=0$ denotes the current frame. The red box denotes the current query of interest $\mathbf{q}_{t}$ or $\mathbf{q}_{t,c}$ in LLSA. Blue boxes plusing the red box indicate keys and values used in attention.
  • Figure 5: Validation of memory efficiency. Test platform: GPU GeForce RTX 2080, Intel® Xeon®(R) Gold 6144 CPU @ 3.50GHz, PyTorch (1.9.0+cu111)
  • ...and 1 more figures