Streaming Speech Recognition with Decoder-Only Large Language Models and Latency Optimization

TL;DR

The paper tackles streaming automatic speech recognition using decoder-only large language models by introducing a MoChA-based read/write policy that dynamically segments streaming audio. It interleaves segment embeddings with prior tokens during training and employs a minimal-latency objective to reduce token-generation delay, all within a unified framework that also supports non-streaming ASR through parameter sharing. Experimental results on AISHELL-1, AISHELL-2, and a multi-domain dataset show strong CER performance for streaming and substantial latency reductions (about 62.5%), with negligible impact on accuracy. The work demonstrates the viability and practicality of end-to-end streaming LLM-based ASR with latency-aware training and a unified training pipeline for both streaming and non-streaming modes.

Abstract

Recent advances have demonstrated the potential of decoderonly large language models (LLMs) for automatic speech recognition (ASR). However, enabling streaming recognition within this framework remains a challenge. In this work, we propose a novel streaming ASR approach that integrates a read/write policy network with monotonic chunkwise attention (MoChA) to dynamically segment speech embeddings. These segments are interleaved with label sequences during training, enabling seamless integration with the LLM. During inference, the audio stream is buffered until the MoChA module triggers a read signal, at which point the buffered segment together with the previous token is fed into the LLM for the next token prediction. We also introduce a minimal-latency training objective to guide the policy network toward accurate segmentation boundaries. Furthermore, we adopt a joint training strategy in which a non-streaming LLM-ASR model and our streaming model share parameters. Experiments on the AISHELL-1 and AISHELL-2 Mandarin benchmarks demonstrate that our method consistently outperforms recent streaming ASR baselines, achieving character error rates of 5.1% and 5.5%, respectively. The latency optimization results in a 62.5% reduction in average token generation delay with negligible impact on recognition accuracy

Figures (2)

• Figure 1: Non-streaming LLM-based ASR architecture. $y_1$ and $y_L$ represent the BOS and EOS token, respectively.
• Figure 2: Our proposed streaming LLM-based ASR architecture. $y_1$ and $y_L$ represent the BOS and EOS token, respectively.