RASST: Fast Cross-modal Retrieval-Augmented Simultaneous Speech Translation

TL;DR

RASST addresses the challenge of translating domain-specific terminology in simultaneous speech translation by tightly integrating a lightweight cross-modal glossary retriever with a Speech LLM. It introduces sliding-window cross-modal retrieval, a data synthesis pipeline for term-aware training, and a robust inference protocol with low overhead, enabling the model to leverage retrieved terms and decide when to apply them during incremental generation. On the ACL 60/60 dev set across En→Zh/De/Ja, RASST achieves up to 16% termination accuracy improvements and up to 3 BLEU point gains, demonstrating effectiveness beyond offline or retrieval-free SST baselines. The approach offers practical benefits for real-time translation systems by combining accurate terminology handling with streaming efficiency, paving the way for deployment with external glossaries and domain-adaptive retrieval.

Abstract

Simultaneous speech translation (SST) produces target text incrementally from partial speech input. Recent speech large language models (Speech LLMs) have substantially improved SST quality, yet they still struggle to correctly translate rare and domain-specific terminology. While retrieval augmentation has been effective for terminology translation in machine translation, bringing retrieval to SST is non-trivial: it requires fast and accurate cross-modal (speech-to-text) retrieval under partial, continually arriving input, and the model must decide whether and when to apply retrieved terms during incremental generation. We propose Retrieval-Augmented Simultaneous Speech Translation (RASST), which tightly integrates cross-modal retrieval into the SST pipeline. RASST trains a lightweight speech-text retriever and performs efficient sliding-window retrieval, providing chunkwise terminology hints to the Speech LLM. We further synthesize training data that teaches the Speech LLM to leverage retrieved terms precisely. Experiments on three language directions of the ACL 60/60 dev set show that RASST improves terminology translation accuracy by up to 16% and increases overall translation quality by up to 3 BLEU points, with ablations confirming the contribution of each component.

Figures (6)

• Figure 1: RASST architecture. The retriever retrieves a set of glossary terms $\hat{G}_i$ for each speech chunk $s_i$. Conditioned on the streaming speech features $f_i$, the retrieved terms $\hat{G}_i$, and the previously generated translations $\hat{y}_{<i}$, the LLM generates the next partial translation $\hat{y}_i$.
• Figure 2: Retriever inference procedure. We encode each glossary term $e_i$ into a feature vector $f_i^e$ and build a FAISS index. At inference time, we slide an speech window of length $W$ with stride $\delta$ (where $\delta < l$, the speech-chunk size) over the incoming speech chunks. For each window, we compute its speech feature and retrieve the Top-$K_1$ nearest glossary terms by cosine similarity. We then aggregate the candidates from all windows within the newly arrived chunk and keep the Top-$K_2$ terms with highest cosine similarities.
• Figure 3: Main results using the tagged glossary. Comparison of Terminology Accuracy (top) and BLEU score (bottom) against latency (StreamLAAL) for En-Zh, En-De, and En-Ja. RASST (red) consistently surpasses the InfiniSST baseline (blue) in both terminology recall and general translation quality across all languages.
• Figure 4: Main results using the paper-extracted glossary. Comparison of Terminology Accuracy (top) and BLEU score (bottom) against latency (StreamLAAL) for En-Zh and En-De. RASST (red) significantly outperforms the InfiniSST baseline (blue) across all latency regimes, especially in terminology translation accuracy.
• Figure 5: The computation overhead of RASST.