Spiralformer: Low Latency Encoder for Streaming Speech Recognition with Circular Layer Skipping and Early Exiting

TL;DR

The paper tackles encoder-side latency in streaming ASR by addressing blockwise processing delays. It introduces Spiralformer, which merges circular layer skipping, layer dropping, cached cross-block computation, and early exiting to reduce system word emission delay while maintaining comparable WER and compute to traditional block-based Transformers. Experimental results on LibriSpeech and CSJ show substantial SWD reductions (e.g., ~21.6% at P50 on LibriSpeech and ~7.0% on CSJ) and notable improvements in maximum theoretical latency, without sacrificing accuracy. This approach offers a practical path toward lower-latency, real-time streaming ASR without prohibitive increases in computation.

Abstract

For streaming speech recognition, a Transformer-based encoder has been widely used with block processing. Although many studies addressed improving emission latency of transducers, little work has been explored for improving encoding latency of the block processing. We seek to reduce latency by frequently emitting a chunk with a small shift rather than scarce large-chunk emissions, resulting in higher computational costs. To efficiently compute with the small chunk shift, we propose a new encoder, Spiralformer, tailored for block processing by combining layer dropping and early exiting. We skip layer computation in a cyclic manner and shift the computed layer in each block spirally, which completes computation for all the layers over the block processing. Experimentally, we observed that our method achieved 21.6% reduction in the averaged token emission delay in Librispeech, and 7.0% in CSJ, compared with the baseline with similar computational cost and word error rates.

Figures (7)

• Figure 1: Delays in blockwise processing. $N_l$ is the number of left context, $N_c$ is the number of non-overlapping chunks, and $N_r$ is the number of right context (look-ahead frames). Internal Word emission Delay (IWD) is a delay of internal modeling, and System Word emission Delays (SWD) is an overall delay of the entire system. Although CTC emissions are aligned with the audio input, the block processing hinders the emission at most the shift size and right context ($N_c+N_r$).
• Figure 2: Circular layer skipping. Left is each block computation; right is overlapped four blocks. The computed results are cached and reused in the next block to complete ordinary full-layer computation. This is an example of a number of total layers $I=12$ and a skipping pitch $p=4$.
• Figure 3: Blockwise processing. $N_l$ is the number of left context, $N_c$ is the number of non-overlapping chunks, and $N_r$ is the number of right context (look-ahead frames). In this example, $\{N_l, N_c, N_r\}=\{1,2,1\}$ and the encoder layers emit $N_c$-frame output $H_b$ for block $b$.
• Figure 4: Layer computation with circular skipping. The $i$-th layer recurrently combines the pre-computed intermediate output of the $(i-1)$-th layer in the previous block $(b-1)$ and of the $(i-p)$-th layer in the current block ($b$).
• Figure 5: Example of computation accumulation. In this case, $\{N_l,N_c,N_r\}=\{3,1,2\}$ and current computing layers are ${\mathcal{C}} = \{1,5,9\}$. The layer skipping pitch is $p=4$. Because of the recurrent process of layer computation in each block, all the circled frames are considered for current block computation colored in red.