CMU's IWSLT 2024 Simultaneous Speech Translation System

TL;DR

This work presents CMU's submission to the IWSLT 2024 SST task for streaming English→German translation, built as an end-to-end system using a WavLM-based speech encoder, a modality adapter, and the Llama2-7B-Base decoder. The authors train in two stages on MuST-C v2 to align speech and text representations before full fine-tuning, and adapt the offline model to SST via a fixed hold-n policy with incremental beam search. Results show offline SacreBLEU of 31.15 and SST SacreBLEU of 29.5 under 2 seconds latency, with AL around 1.96 s and LAAL around 2.22 s, indicating substantial latency improvements at a modest BLEU cost. Ablation studies reveal the benefits of a strong WavLM encoder and the trade-offs between different LLMs, with overfitting concerns addressed by learning-rate adjustments during the second stage. The work demonstrates a viable, scalable approach to end-to-end SST that leverages powerful speech encoders and LLMs for streaming translation.

Abstract

This paper describes CMU's submission to the IWSLT 2024 Simultaneous Speech Translation (SST) task for translating English speech to German text in a streaming manner. Our end-to-end speech-to-text (ST) system integrates the WavLM speech encoder, a modality adapter, and the Llama2-7B-Base model as the decoder. We employ a two-stage training approach: initially, we align the representations of speech and text, followed by full fine-tuning. Both stages are trained on MuST-c v2 data with cross-entropy loss. We adapt our offline ST model for SST using a simple fixed hold-n policy. Experiments show that our model obtains an offline BLEU score of 31.1 and a BLEU score of 29.5 under 2 seconds latency on the MuST-C-v2 tst-COMMON.

Figures (1)

• Figure 1: Offline ST model architecture based on WavLM encoder and Llama2 7B decoder.