Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset

Ashish V. Thapliyal, Jordi Pont-Tuset, Xi Chen, Radu Soricut

TL;DR

XM3600 introduces a massively multilingual image captioning benchmark with 3600 images and captions across 36 languages, designed to avoid translation artifacts through a cross-language annotation protocol and geographically diverse image sampling from Open Images. The dataset enables robust model ranking by correlating automated metrics with human judgments, demonstrating that XM3600 references yield high alignment with human evaluations and outperform translated silver standards. The paper details language and image selection, annotation methodology, and extensive model comparisons using Transformer-based multimodal architectures, confirming XM3600 as a practical, scalable benchmark for multilingual captioning research. Limitations include potential language and region biases and constraints on data release, but XM3600 is positioned to accelerate fair, cross-language progress and accessibility implications for visually impaired users worldwide.

Abstract

Research in massively multilingual image captioning has been severely hampered by a lack of high-quality evaluation datasets. In this paper we present the Crossmodal-3600 dataset (XM3600 in short), a geographically diverse set of 3600 images annotated with human-generated reference captions in 36 languages. The images were selected from across the world, covering regions where the 36 languages are spoken, and annotated with captions that achieve consistency in terms of style across all languages, while avoiding annotation artifacts due to direct translation. We apply this benchmark to model selection for massively multilingual image captioning models, and show superior correlation results with human evaluations when using XM3600 as golden references for automatic metrics.

Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset

TL;DR

XM3600 introduces a massively multilingual image captioning benchmark with 3600 images and captions across 36 languages, designed to avoid translation artifacts through a cross-language annotation protocol and geographically diverse image sampling from Open Images. The dataset enables robust model ranking by correlating automated metrics with human judgments, demonstrating that XM3600 references yield high alignment with human evaluations and outperform translated silver standards. The paper details language and image selection, annotation methodology, and extensive model comparisons using Transformer-based multimodal architectures, confirming XM3600 as a practical, scalable benchmark for multilingual captioning research. Limitations include potential language and region biases and constraints on data release, but XM3600 is positioned to accelerate fair, cross-language progress and accessibility implications for visually impaired users worldwide.

Abstract

Research in massively multilingual image captioning has been severely hampered by a lack of high-quality evaluation datasets. In this paper we present the Crossmodal-3600 dataset (XM3600 in short), a geographically diverse set of 3600 images annotated with human-generated reference captions in 36 languages. The images were selected from across the world, covering regions where the 36 languages are spoken, and annotated with captions that achieve consistency in terms of style across all languages, while avoiding annotation artifacts due to direct translation. We apply this benchmark to model selection for massively multilingual image captioning models, and show superior correlation results with human evaluations when using XM3600 as golden references for automatic metrics.
Paper Structure (20 sections, 4 figures, 11 tables)

This paper contains 20 sections, 4 figures, 11 tables.

Table of Contents

  1. Introduction
  2. The XM3600 Dataset
  3. Language Selection
  4. Image Selection
  5. Caption Annotation
  6. Caption Statistics
  7. Caption Quality
  8. Annotator Details
  9. Model Comparison using XM3600
  10. Datasets
  11. Models
  12. Human Evaluation
  13. Results
  14. Conclusions
  15. Limitations
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Sample captions in three different languages (out of 36 -- see full list of captions in Appendix \ref{['app:additional_captions']}), showcasing the creation of annotations that are consistent in style across languages, while being free of direct-translation artefacts (e.g. the Spanish "number 42" or the Thai "convertibles" would not be possible when directly translating from the English versions).
  • Figure 2: A sample of images in the XM3600 dataset, together with the language for which they have been selected. Overall, the images span regions over 36 different languages and 6 different continents.
  • Figure 3: The architecture for the family of multilingual image captioning models used in the experiments.
  • Figure 4: Example captions in the 36 languages covered in XM3600