Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Survey on Multi-modal Machine Translation: Tasks, Methods and Challenges

Huangjun Shen, Liangying Shao, Wenbo Li, Zhibin Lan, Zhanyu Liu, Jinsong Su

TL;DR

This survey surveys the field of multi-modal machine translation (MMT), focusing on how visual context from images and videos can disambiguate and enrich translation beyond text alone. It categorizes scene-image MMT into model design, training, and analysis, and details four design directions (double attention, images as supplements, text-to-image generation, and text-to-image retrieval), along with training strategies (multi-task, contrastive, unsupervised, and pre-training). It further surveys non-scene-image MMT tasks (e-commerce, TIMT, video-guided MT, multi-modal SiMT, and chat MT), datasets, evaluation metrics, and comparative model performance, highlighting both gains and limitations. The paper concludes with future directions including integration with large language models, development of image-aware evaluation, and the need for large-scale, high-quality multi-domain datasets, underscoring the practical impact of MMT on applications like subtitle translation and cross-border commerce.

Abstract

In recent years, multi-modal machine translation has attracted significant interest in both academia and industry due to its superior performance. It takes both textual and visual modalities as inputs, leveraging visual context to tackle the ambiguities in source texts. In this paper, we begin by offering an exhaustive overview of 99 prior works, comprehensively summarizing representative studies from the perspectives of dominant models, datasets, and evaluation metrics. Afterwards, we analyze the impact of various factors on model performance and finally discuss the possible research directions for this task in the future. Over time, multi-modal machine translation has developed more types to meet diverse needs. Unlike previous surveys confined to the early stage of multi-modal machine translation, our survey thoroughly concludes these emerging types from different aspects, so as to provide researchers with a better understanding of its current state.

A Survey on Multi-modal Machine Translation: Tasks, Methods and Challenges

TL;DR

This survey surveys the field of multi-modal machine translation (MMT), focusing on how visual context from images and videos can disambiguate and enrich translation beyond text alone. It categorizes scene-image MMT into model design, training, and analysis, and details four design directions (double attention, images as supplements, text-to-image generation, and text-to-image retrieval), along with training strategies (multi-task, contrastive, unsupervised, and pre-training). It further surveys non-scene-image MMT tasks (e-commerce, TIMT, video-guided MT, multi-modal SiMT, and chat MT), datasets, evaluation metrics, and comparative model performance, highlighting both gains and limitations. The paper concludes with future directions including integration with large language models, development of image-aware evaluation, and the need for large-scale, high-quality multi-domain datasets, underscoring the practical impact of MMT on applications like subtitle translation and cross-border commerce.

Abstract

In recent years, multi-modal machine translation has attracted significant interest in both academia and industry due to its superior performance. It takes both textual and visual modalities as inputs, leveraging visual context to tackle the ambiguities in source texts. In this paper, we begin by offering an exhaustive overview of 99 prior works, comprehensively summarizing representative studies from the perspectives of dominant models, datasets, and evaluation metrics. Afterwards, we analyze the impact of various factors on model performance and finally discuss the possible research directions for this task in the future. Over time, multi-modal machine translation has developed more types to meet diverse needs. Unlike previous surveys confined to the early stage of multi-modal machine translation, our survey thoroughly concludes these emerging types from different aspects, so as to provide researchers with a better understanding of its current state.
Paper Structure (24 sections, 4 figures, 3 tables)

This paper contains 24 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Scene-Image Multi-modal Machine Translation
  3. Model Design
  4. Double Attention Mechanism
  5. Images as A Supplement to Texts
  6. Text-to-image Generation
  7. Text-to-image Retrieval
  8. Model Training
  9. Multi-task Learning
  10. Contrastive Learning
  11. Unsupervised Learning
  12. Pre-traning
  13. Model Analysis
  14. Other Types of Multi-modal Machine Translation
  15. E-commerce Product-oriented Machine Translation
  16. ...and 9 more sections

Figures (4)

  • Figure 1: An example shows the difference between the conventional text-only MT and MMT models.
  • Figure 2: Paper publications of MMT at the top computer science conferences and journals.
  • Figure 3: The Taxonomy of Representative Studies on MMT.
  • Figure 4: Experimental results of Transformer-based scene-image MMT models. The meanings of symbols in the Techniques column are as follows: adversarial training (AT), back translation (BT), cross-modal filter mechanism (CFM), capsule network (CN), contrastive learning (CT), double attention (DA), gated fusion (GF), graph neural network (GNN), imagination (IG), knowledge distillation (KD), image retrieval (IR), multiple decoding (MD), multitask learning (ML), mix-up (MU), pre-training (PT), reinforcement learning (RL), synthetic images (SI) and unsupervised learning (UL). B/M refers to BLEU/METEOR.