SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translation

TL;DR

This work tackles the challenge of performing simultaneous speech-to-speech translation by enabling speech-capable LLMs to infer in real time. It introduces SimulS2S-LLM, an offline-trained framework that uses boundary-aware speech prompts generated by CIF, a test-time Wait-$k$ policy, and an incremental beam search to produce discrete target-speech tokens that are vocoded into output speech. Key contributions include boundary-aware prompt extraction, multi-layer hidden-state fusion for robust token prediction, and an LM-assisted streaming decoder, all validated on the CVSS-C corpus where the method surpasses strong baselines in quality-latency trade-offs. The approach demonstrates the practical viability of leveraging LLMs for simultaneous S2ST and sets a foundation for future improvements in streaming efficiency and broader language support.

Abstract

Simultaneous speech translation (SST) outputs translations in parallel with streaming speech input, balancing translation quality and latency. While large language models (LLMs) have been extended to handle the speech modality, streaming remains challenging as speech is prepended as a prompt for the entire generation process. To unlock LLM streaming capability, this paper proposes SimulS2S-LLM, which trains speech LLMs offline and employs a test-time policy to guide simultaneous inference. SimulS2S-LLM alleviates the mismatch between training and inference by extracting boundary-aware speech prompts that allows it to be better matched with text input data. SimulS2S-LLM achieves simultaneous speech-to-speech translation (Simul-S2ST) by predicting discrete output speech tokens and then synthesising output speech using a pre-trained vocoder. An incremental beam search is designed to expand the search space of speech token prediction without increasing latency. Experiments on the CVSS speech data show that SimulS2S-LLM offers a better translation quality-latency trade-off than existing methods that use the same training data, such as improving ASR-BLEU scores by 3 points at similar latency.

Figures (11)

• Figure 1: Illustration of SimulS2S-LLM offline training. $\bm{\oplus}$ denotes addition. Prefix and postfix text contain template instructions. The hidden states of each LLM layer are weighted ($\beta_i$) and summed.
• Figure 2: Illustration of the boundary-aware speech prompt extraction using the CIF. $\bm{\oplus}$ denotes addition.
• Figure 3: Illustration of the simultaneous inference of offline-trained SimulS2S-LLM (with wait-3 as an example) (a) the generation of the $1$-st hidden state and corresponding speech tokens $t_i$; (b) $2$-nd generation; (c) overall illustration of the hidden state generation order according to the speech prompt (always wait 3 more steps here).
• Figure 5: Simul-S2TT quality-latency trade-off curves on CVSS-C Es-En test set. The x-axis represents the latency, measured by AL, and the y-axis represents the speech translation quality, measured by BLEU.
• Figure 6: Simul-S2ST Es-En results for SimulS2S-LLM with single or multi-layer LLM hidden states.