m3P: Towards Multimodal Multilingual Translation with Multimodal Prompt

TL;DR

A framework to leverage the multimodal prompt to guide the Multimodal Multilingual Neural Machine Translation (m3P), which aligns the representations of different languages with the same meaning and generates the conditional vision-language memory for translation.

Abstract

Multilingual translation supports multiple translation directions by projecting all languages in a shared space, but the translation quality is undermined by the difference between languages in the text-only modality, especially when the number of languages is large. To bridge this gap, we introduce visual context as the universal language-independent representation to facilitate multilingual translation. In this paper, we propose a framework to leverage the multimodal prompt to guide the Multimodal Multilingual neural Machine Translation (m3P), which aligns the representations of different languages with the same meaning and generates the conditional vision-language memory for translation. We construct a multilingual multimodal instruction dataset (InstrMulti102) to support 102 languages. Our method aims to minimize the representation distance of different languages by regarding the image as a central language. Experimental results show that m3P outperforms previous text-only baselines and multilingual multimodal methods by a large margin. Furthermore, the probing experiments validate the effectiveness of our method in enhancing translation under the low-resource and massively multilingual scenario.

Figures (7)

• Figure 1: Comparison between (a) the bilingual translation baseline and (b) our proposed m$^3$P.
• Figure 2: Overview of our method. $s^{k}=\{s_u^k\}_{u=1}^{U}$ denotes the representations of the source sentence of $U$ tokens. We reshape the original image $z^k \in \mathcal{R}^{H \times W \times C}$ into $V$ patches and then encoded as $h^k=\{s_v^k\}_{v=1}^{V}$ with the vision Transformer. Given the source and visual representations $s^k$ and $h^k$, the multilingual multimodal contrastive learning (MMCL) adopted to minimize the distance between $s^k$ of different languages and $h^k$, which greatly encourages multilingual multimodal agreement in a shared space. Conditioned on the image tokens as (key,value), the language features as the query attend the multi-head attention to generate final encoder states $e^k=\{e_u^k\}_{u=1}^{U}$ as conditional vision-language for multilingual translation.
• Figure 3: Multimodal prompt for LLM.
• Figure 4: Visualization of the sentence average encoder representations of all languages from the multilingual baseline (a) and our multilingual model supervised by the image context (b). Each color denotes one language.
• Figure 5: The performance of our method on Flickr2016 (a) En$\to$fr, (b) En$\to$De, (c) Fr$\to$En, and (d) De$\to$En with different sizes of training data on Flickr2016.