Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SimulSeamless: FBK at IWSLT 2024 Simultaneous Speech Translation

Sara Papi, Marco Gaido, Matteo Negri, Luisa Bentivogli

TL;DR

SimulST requires balancing translation quality and latency in live scenarios. The authors repurpose an off-the-shelf SeamlessM4T model using the AlignAtt cross-attention policy to enable simultaneous translation without additional training. Their SimulSeamless system achieves competitive results across multiple language pairs in IWSLT 2024, while covering more than 143 source languages and 200 target languages. The work provides a generic, readily deployable approach and releases the code under Apache 2.0.

Abstract

This paper describes the FBK's participation in the Simultaneous Translation Evaluation Campaign at IWSLT 2024. For this year's submission in the speech-to-text translation (ST) sub-track, we propose SimulSeamless, which is realized by combining AlignAtt and SeamlessM4T in its medium configuration. The SeamlessM4T model is used "off-the-shelf" and its simultaneous inference is enabled through the adoption of AlignAtt, a SimulST policy based on cross-attention that can be applied without any retraining or adaptation of the underlying model for the simultaneous task. We participated in all the Shared Task languages (English->{German, Japanese, Chinese}, and Czech->English), achieving acceptable or even better results compared to last year's submissions. SimulSeamless, covering more than 143 source languages and 200 target languages, is released at: https://github.com/hlt-mt/FBK-fairseq/.

SimulSeamless: FBK at IWSLT 2024 Simultaneous Speech Translation

TL;DR

SimulST requires balancing translation quality and latency in live scenarios. The authors repurpose an off-the-shelf SeamlessM4T model using the AlignAtt cross-attention policy to enable simultaneous translation without additional training. Their SimulSeamless system achieves competitive results across multiple language pairs in IWSLT 2024, while covering more than 143 source languages and 200 target languages. The work provides a generic, readily deployable approach and releases the code under Apache 2.0.

Abstract

This paper describes the FBK's participation in the Simultaneous Translation Evaluation Campaign at IWSLT 2024. For this year's submission in the speech-to-text translation (ST) sub-track, we propose SimulSeamless, which is realized by combining AlignAtt and SeamlessM4T in its medium configuration. The SeamlessM4T model is used "off-the-shelf" and its simultaneous inference is enabled through the adoption of AlignAtt, a SimulST policy based on cross-attention that can be applied without any retraining or adaptation of the underlying model for the simultaneous task. We participated in all the Shared Task languages (English->{German, Japanese, Chinese}, and Czech->English), achieving acceptable or even better results compared to last year's submissions. SimulSeamless, covering more than 143 source languages and 200 target languages, is released at: https://github.com/hlt-mt/FBK-fairseq/.
Paper Structure (10 sections, 4 figures, 1 table)

This paper contains 10 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. SimulSeamless
  3. SeamlessM4T.
  4. AlignAtt.
  5. SeamlessM4T + AlignAtt = SimulSeamless.
  6. Experimental Settings
  7. Results
  8. Submission Selection
  9. Comparison with Last Year's Participants
  10. Conclusions

Figures (4)

  • Figure 1: Representation of the SeamlessM4T model combined with AlignAtt: SimulSeamless.
  • Figure 2: Example of skewed cross-attention scores representation towards some frames.
  • Figure 3: Translation quality (BLEU$\uparrow$) scores of SimulSeamless on MuST-C v2.0 tst-COMMON for English (en) to German (de), Japanese (ja), and Chinese (zh), and on the IWSLT 2024 dev set for Czech (cs) to English by varying the decoder layer from which cross-attention scores are extracted from.
  • Figure 4: Latency (AL$\downarrow$) scores of SimulSeamless on MuST-C v2.0 tst-COMMON for English (en) to German (de), Japanese (ja), and Chinese (zh), and on the IWSLT 2024 dev set for Czech (cs) to English by varying the decoder layer from which cross-attention scores are extracted from.