Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Zero-Shot Multimodal Machine Translation

Matthieu Futeral, Cordelia Schmid, Benoît Sagot, Rachel Bawden

TL;DR

This work tackles the data bottleneck in multimodal machine translation by introducing ZeroMMT, a zero-shot approach that repurposes a strong text-only MT model to leverage English image-caption data for translating into multiple languages without requiring parallel multimodal data. It jointly optimizes visually conditioned masked language modelling and KL-divergence-based regularization to preserve translation ability while exploiting visual context for disambiguation. The method extends CoMMuTE to Arabic, Russian, and Chinese, demonstrates competitive disambiguation performance close to fully supervised models, and enables inference-time control of the disambiguation-generation trade-off via classifier-free guidance. By enabling zero-shot deployment and providing public code and data, ZeroMMT broadens the reach of MMT to languages with limited multimodal resources and paves the way for adaptable, visually-informed translation systems.

Abstract

Current multimodal machine translation (MMT) systems rely on fully supervised data (i.e models are trained on sentences with their translations and accompanying images). However, this type of data is costly to collect, limiting the extension of MMT to other language pairs for which such data does not exist. In this work, we propose a method to bypass the need for fully supervised data to train MMT systems, using multimodal English data only. Our method, called ZeroMMT, consists in adapting a strong text-only machine translation (MT) model by training it on a mixture of two objectives: visually conditioned masked language modelling and the Kullback-Leibler divergence between the original and new MMT outputs. We evaluate on standard MMT benchmarks and the recently released CoMMuTE, a contrastive benchmark aiming to evaluate how well models use images to disambiguate English sentences. We obtain disambiguation performance close to state-of-the-art MMT models trained additionally on fully supervised examples. To prove that our method generalizes to languages with no fully supervised training data available, we extend the CoMMuTE evaluation dataset to three new languages: Arabic, Russian and Chinese. We further show that we can control the trade-off between disambiguation capabilities and translation fidelity at inference time using classifier-free guidance and without any additional data. Our code, data and trained models are publicly accessible.

Towards Zero-Shot Multimodal Machine Translation

TL;DR

This work tackles the data bottleneck in multimodal machine translation by introducing ZeroMMT, a zero-shot approach that repurposes a strong text-only MT model to leverage English image-caption data for translating into multiple languages without requiring parallel multimodal data. It jointly optimizes visually conditioned masked language modelling and KL-divergence-based regularization to preserve translation ability while exploiting visual context for disambiguation. The method extends CoMMuTE to Arabic, Russian, and Chinese, demonstrates competitive disambiguation performance close to fully supervised models, and enables inference-time control of the disambiguation-generation trade-off via classifier-free guidance. By enabling zero-shot deployment and providing public code and data, ZeroMMT broadens the reach of MMT to languages with limited multimodal resources and paves the way for adaptable, visually-informed translation systems.

Abstract

Current multimodal machine translation (MMT) systems rely on fully supervised data (i.e models are trained on sentences with their translations and accompanying images). However, this type of data is costly to collect, limiting the extension of MMT to other language pairs for which such data does not exist. In this work, we propose a method to bypass the need for fully supervised data to train MMT systems, using multimodal English data only. Our method, called ZeroMMT, consists in adapting a strong text-only machine translation (MT) model by training it on a mixture of two objectives: visually conditioned masked language modelling and the Kullback-Leibler divergence between the original and new MMT outputs. We evaluate on standard MMT benchmarks and the recently released CoMMuTE, a contrastive benchmark aiming to evaluate how well models use images to disambiguate English sentences. We obtain disambiguation performance close to state-of-the-art MMT models trained additionally on fully supervised examples. To prove that our method generalizes to languages with no fully supervised training data available, we extend the CoMMuTE evaluation dataset to three new languages: Arabic, Russian and Chinese. We further show that we can control the trade-off between disambiguation capabilities and translation fidelity at inference time using classifier-free guidance and without any additional data. Our code, data and trained models are publicly accessible.
Paper Structure (30 sections, 3 equations, 10 figures, 16 tables)

This paper contains 30 sections, 3 equations, 10 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Training MMT systems
  4. Evaluating MMT systems
  5. Extending the CoMMuTE benchmark
  6. Our Approach
  7. Experiments
  8. Data
  9. Implementation details
  10. Modelling
  11. Training
  12. Evaluation
  13. Results
  14. Baselines and comparative models
  15. Quantitative results
  16. ...and 15 more sections

Figures (10)

  • Figure 1: Overview of our approach. We train on two objectives: Visually conditioned masked language modelling (VMLM) and Kullback-Leibler (KL) divergence. All weights are frozen during training except the visual projector and the adapters in the MT model.
  • Figure 2: Translations of CoMMuTE by our approach, ZeroMMT-600M, and the NLLB distilled MT model.
  • Figure 3: French human evaluation on 100 examples from CoMMuTE. 'Draw' means both translations are exactly the same or considered of same quality.
  • Figure 4: French human evaluation on 100 examples from M30K test sets. 'Draw' means both translations are exactly the same or considered of the same quality.
  • Figure 5: Chinese human evaluation on 100 examples from CoMMuTE. 'Draw' means both translations are exactly the same or considered of same quality.
  • ...and 5 more figures