3AM: An Ambiguity-Aware Multi-Modal Machine Translation Dataset

TL;DR

This work targets a core limitation in multimodal machine translation: existing datasets underutilize visual information due to shallow ambiguity. It introduces 3AM, an ambiguity-aware English-Chinese MMT dataset with 26K image-caption pairs designed to force models to ground translation in visual content, using a BabelNet-based word sense dictionary and a WSD model to select ambiguous instances. Across text-only and multimodal baselines, models trained on 3AM demonstrate significantly improved visual awareness and disambiguation performance, suggesting that 3AM better elicits genuine multimodal understanding. The dataset, along with its accompanying code and annotations, provides a valuable resource to advance rigorous evaluation of visual grounding in translation and broader multimodal learning.

Abstract

Multimodal machine translation (MMT) is a challenging task that seeks to improve translation quality by incorporating visual information. However, recent studies have indicated that the visual information provided by existing MMT datasets is insufficient, causing models to disregard it and overestimate their capabilities. This issue presents a significant obstacle to the development of MMT research. This paper presents a novel solution to this issue by introducing 3AM, an ambiguity-aware MMT dataset comprising 26,000 parallel sentence pairs in English and Chinese, each with corresponding images. Our dataset is specifically designed to include more ambiguity and a greater variety of both captions and images than other MMT datasets. We utilize a word sense disambiguation model to select ambiguous data from vision-and-language datasets, resulting in a more challenging dataset. We further benchmark several state-of-the-art MMT models on our proposed dataset. Experimental results show that MMT models trained on our dataset exhibit a greater ability to exploit visual information than those trained on other MMT datasets. Our work provides a valuable resource for researchers in the field of multimodal learning and encourages further exploration in this area. The data, code and scripts are freely available at https://github.com/MaxyLee/3AM.

Figures (5)

• Figure 1: Examples of our 3AM dataset, where ambiguous words are shown in bold, with red and blue indicating its incorrect and correct translation respectively. For instance, the word 'palm' has two different meanings: "palm tree" and "palm of the hand". Only from the image can it be distinguished that the correct meaning is the former.
• Figure 2: The process for constructing the 3AM dataset involves several steps. Firstly, we extract ambiguous words from existing WSD datasets and create a word sense dictionary using BabelNet. We then use this dictionary to filter sentences that contain ambiguous words and score them using a WSD model. Finally, we rank the sentences according to their scores to obtain the ambiguous data.
• Figure 3: Statistical histogram distributions on Multi30K, MSCTD, and 3AM. Compared with other datasets, 3AM contains longer captions with more unique nouns and verbs and higher ambiguity scores.
• Figure 4: Plot of the most common words in the captions of Multi30K and 3AM, the words in the 3AM dataset are more evenly distributed.
• Figure 5: A case study of the 3AM dataset. The ambiguous word in the source sentence is in bold. The red and blue represent incorrectly and correctly translated words respectively.