Navigating the Minefield of MT Beam Search in Cascaded Streaming Speech Translation

TL;DR

The paper tackles the problem of adapting beam search to simultaneous MT within cascaded real-time speech translation, where ASR outputs (including incomplete words and intermediate results) must be translated with minimal latency. It introduces a streaming beam-search framework that maintains state across incremental reads, emits both intermediate and final outputs, and handles incomplete ASR tokens, while using length-normalized token probabilities (excluding the MILK-derived write probability) for beam scoring. Key contributions include state management via KV-cache, careful handling of intermediate inputs and sentence boundaries, and batching approaches to improve efficiency on constrained hardware. Empirical results on embedded devices show the streaming beam search yields a BLEU improvement of about 1 point over greedy decoding, up to 40% CPU-time reductions, and more than 20% reduction in character flicker compared with a repeated-retranslation baseline, demonstrating practical benefits for real-time cascaded speech translation.

Abstract

We adapt the well-known beam-search algorithm for machine translation to operate in a cascaded real-time speech translation system. This proved to be more complex than initially anticipated, due to four key challenges: (1) real-time processing of intermediate and final transcriptions with incomplete words from ASR, (2) emitting intermediate and final translations with minimal user perceived latency, (3) handling beam search hypotheses that have unequal length and different model state, and (4) handling sentence boundaries. Previous work in the field of simultaneous machine translation only implemented greedy decoding. We present a beam-search realization that handles all of the above, providing guidance through the minefield of challenges. Our approach increases the BLEU score by 1 point compared to greedy search, reduces the CPU time by up to 40% and character flicker rate by 20+% compared to a baseline heuristic that just retranslates input repeatedly.