Moonshine v2: Ergodic Streaming Encoder ASR for Latency-Critical Speech Applications

TL;DR

Moonshine v2 tackles latency-critical on-device ASR by replacing full-attention encoders with ergodic streaming encoders that use sliding-window self-attention. This design bounds time-to-first-token (TTFT) while preserving competitive accuracy, enabling real-time, edge-allowed transcription. The approach achieves state-of-the-art results for compact models, matching larger architectures at a fraction of compute and latency, and demonstrates favorable latency comparisons against Whisper on edge hardware. The work highlights the practicality of local attention for interactive speech interfaces and outlines avenues for end-to-end streaming decoding and multilingual expansion.

Abstract

Latency-critical speech applications (e.g., live transcription, voice commands, and real-time translation) demand low time-to-first-token (TTFT) and high transcription accuracy, particularly on resource-constrained edge devices. Full-attention Transformer encoders remain a strong accuracy baseline for automatic speech recognition (ASR) because every frame can directly attend to every other frame, which resolves otherwise locally ambiguous acoustics using distant lexical context. However, this global dependency incurs quadratic complexity in sequence length, inducing an inherent "encode-the-whole-utterance" latency profile. For streaming use cases, this causes TTFT to grow linearly with utterance length as the encoder must process the entire prefix before any decoder token can be emitted. To better meet the needs of on-device, streaming ASR use cases we introduce Moonshine v2, an ergodic streaming-encoder ASR model that employs sliding-window self-attention to achieve bounded, low-latency inference while preserving strong local context. Our models achieve state of the art word error rates across standard benchmarks, attaining accuracy on-par with models 6x their size while running significantly faster. These results demonstrate that carefully designed local attention is competitive with the accuracy of full attention at a fraction of the size and latency cost, opening new possibilities for interactive speech interfaces on edge devices.

Figures (5)

• Figure 1: Illustrative time-to-first-token (TTFT) for a full-attention encoder as a function of audio length, for processors with different peak throughput (TOPS). The estimate includes both a linear non-attention term and a quadratic self-attention mixing term. The dotted horizontal line shows a 0.1 TOPS sliding-window encoder with $w=20$ frames. The dashed line indicates a 250 ms one-way delay limit often used as a practical upper bound for acceptable interactive voice in private networks cisco_delay_details.
• Figure 2: Conceptual TTFT timelines. In full attention, encoding begins after the entire audio has arrived. With sliding-window attention, encoding proceeds incrementally and overlaps with audio capture, so the remaining work after the last chunk is smaller.
• Figure 3: Audio preprocessor overview with tensor shapes, single-example. Dimensions $T = \lfloor \frac{N}{80} \rfloor$, $T_1 = \lceil \frac{T}{2} \rceil$, and $T_2 = \lceil \frac{T_1}{2} \rceil$
• Figure 4: Accuracy vs. parameter count on Open ASR leaderboard averages.
• Figure 5: Time to first token (TTFT) versus input audio duration for Moonshine and Moonshine v2 models. The original Moonshine models use a full-attention encoder, which results in TTFT latency that grows with input audio duration. Sliding window attention in the Moonshine v2 encoder results in a fixed encoding latency, regardless of audio duration.