EMMeTT: Efficient Multimodal Machine Translation Training

TL;DR

This paper tackles efficient joint multimodal training for translation within Speech-LLMs, addressing catastrophic forgetting and the need to preserve text translation while acquiring automatic speech translation. It introduces EMMeTT, a framework that jointly trains NMT and AST on two architectures (SALM-T5 and BESTOW-GPT) using a Canary-1B speech encoder, with stationary data blending, 2D bucketing, and a batch-size optimizer. Empirical results on FLORES and FLEURS subsets across English, French, German, and Spanish show that multimodal training improves speech translation and maintains text NMT performance, outperforming AST baselines. Efficiency gains enable faster training and better memory usage, suggesting the approach scales to larger models and more modalities; the authors release the code within NVIDIA NeMo.

Abstract

A rising interest in the modality extension of foundation language models warrants discussion on the most effective, and efficient, multimodal training approach. This work focuses on neural machine translation (NMT) and proposes a joint multimodal training regime of Speech-LLM to include automatic speech translation (AST). We investigate two different foundation model architectures, decoder-only GPT and encoder-decoder T5, extended with Canary-1B's speech encoder. To handle joint multimodal training, we propose a novel training framework called EMMeTT. EMMeTT improves training efficiency with the following: balanced sampling across languages, datasets, and modalities; efficient sequential data iteration; and a novel 2D bucketing scheme for multimodal data, complemented by a batch size optimizer (OOMptimizer). We show that a multimodal training consistently helps with both architectures. Moreover, SALM-T5 trained with EMMeTT retains the original NMT capability while outperforming AST baselines on four-language subsets of FLORES and FLEURS. The resultant Multimodal Translation Model produces strong text and speech translation results at the same time.

Figures (1)

• Figure 1: (a) SALM-T5 architecture. (b) BESTOW-GPT architecture. The proposed multimodal joint training is visualized on both architectures where green shade denotes speech translation training and blue dotted shade denotes text translation training.