StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning

TL;DR

StreamSpeech tackles the challenge of simultaneous speech-to-speech translation by proposing an all-in-one, end-to-end model that jointly learns translation and the READ/WRITE policy under a multi-task framework. It uses a two-pass architecture—autoregressive speech-to-text translation to obtain target-text representations $D^{text}$, followed by non-autoregressive text-to-unit synthesis for target speech—guided by CTC-based alignments, and trained with four interrelated losses. Empirical results on CVSS-C show state-of-the-art performance for both offline S2ST and Simul-S2ST, while maintaining the ability to display high-quality intermediate ASR and S2TT outputs during inference. The work also demonstrates robust latency adaptability through multi-chunk training and ablations, emphasizing the importance of alignment-guided policy in achieving coherent, low-latency streaming translation with minimal error propagation.

Abstract

Simultaneous speech-to-speech translation (Simul-S2ST, a.k.a streaming speech translation) outputs target speech while receiving streaming speech inputs, which is critical for real-time communication. Beyond accomplishing translation between speech, Simul-S2ST requires a policy to control the model to generate corresponding target speech at the opportune moment within speech inputs, thereby posing a double challenge of translation and policy. In this paper, we propose StreamSpeech, a direct Simul-S2ST model that jointly learns translation and simultaneous policy in a unified framework of multi-task learning. Adhering to a multi-task learning approach, StreamSpeech can perform offline and simultaneous speech recognition, speech translation and speech synthesis via an "All-in-One" seamless model. Experiments on CVSS benchmark demonstrate that StreamSpeech achieves state-of-the-art performance in both offline S2ST and Simul-S2ST tasks. Besides, StreamSpeech is able to present high-quality intermediate results (i.e., ASR or translation results) during simultaneous translation process, offering a more comprehensive real-time communication experience.

Figures (10)

• Figure 1: StreamSpeech is an "All in One" seamless model for multiple offline and simultaneous tasks.
• Figure 2: StreamSpeech employs two-pass architecture that first converts source speech into target text hidden states $D^{text}$ (autoregressive speech-to-text translation, AR-S2TT) and then generates target speech via non-autoregressive text-to-unit generation. The source/target/unit CTC decoders are introduced to learn alignments via multiple tasks of speech recognition (ASR), non-autoregressive speech-to-text translation (NAR-S2TT) and speech-to-unit translation (S2UT), accordingly guiding StreamSpeech when to start recognizing, translating and synthesizing.
• Figure 3: Architecture of chunk-based Conformer.
• Figure 4: Simul-S2ST results (quality against latency) on CVSS-C Fr$\rightarrow$En, Es$\rightarrow$En, De$\rightarrow$En test sets. The hollow points represent computation-aware latency, which includes the inference time consumed by the model. Some simultaneous outputs of StreamSpeech can be heard at https://ictnlp.github.io/StreamSpeech-site/.
• Figure 5: Comparison of direct and cascaded Simul-S2ST systems.