Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MIT-10M: A Large Scale Parallel Corpus of Multilingual Image Translation

Bo Li, Shaolin Zhu, Lijie Wen

TL;DR

MIT-10M addresses the scarcity of large-scale real-world multilingual image translation data by introducing a 10-million image-text parallel corpus spanning 14 languages and 840k high-resolution images. The dataset is produced via a three-stage pipeline: web-scale data collection, precise OCR-based text extraction, and multilingual translation with automated and human validation, achieving a translation accuracy of 99.4% on human-evaluated samples. Extensive experiments compare cascaded and end-to-end IT models, revealing superior performance of end-to-end approaches on MIT-10M, and show that fine-tuning with MIT-10M yields substantial gains across BLEU, chrF++, and METEOR. Overall, MIT-10M provides a realistic, diverse, and scalable benchmark that enhances training, evaluation, and generalization for multilingual image translation in real-world scenarios.

Abstract

Image Translation (IT) holds immense potential across diverse domains, enabling the translation of textual content within images into various languages. However, existing datasets often suffer from limitations in scale, diversity, and quality, hindering the development and evaluation of IT models. To address this issue, we introduce MIT-10M, a large-scale parallel corpus of multilingual image translation with over 10M image-text pairs derived from real-world data, which has undergone extensive data cleaning and multilingual translation validation. It contains 840K images in three sizes, 28 categories, tasks with three levels of difficulty and 14 languages image-text pairs, which is a considerable improvement on existing datasets. We conduct extensive experiments to evaluate and train models on MIT-10M. The experimental results clearly indicate that our dataset has higher adaptability when it comes to evaluating the performance of the models in tackling challenging and complex image translation tasks in the real world. Moreover, the performance of the model fine-tuned with MIT-10M has tripled compared to the baseline model, further confirming its superiority.

MIT-10M: A Large Scale Parallel Corpus of Multilingual Image Translation

TL;DR

MIT-10M addresses the scarcity of large-scale real-world multilingual image translation data by introducing a 10-million image-text parallel corpus spanning 14 languages and 840k high-resolution images. The dataset is produced via a three-stage pipeline: web-scale data collection, precise OCR-based text extraction, and multilingual translation with automated and human validation, achieving a translation accuracy of 99.4% on human-evaluated samples. Extensive experiments compare cascaded and end-to-end IT models, revealing superior performance of end-to-end approaches on MIT-10M, and show that fine-tuning with MIT-10M yields substantial gains across BLEU, chrF++, and METEOR. Overall, MIT-10M provides a realistic, diverse, and scalable benchmark that enhances training, evaluation, and generalization for multilingual image translation in real-world scenarios.

Abstract

Image Translation (IT) holds immense potential across diverse domains, enabling the translation of textual content within images into various languages. However, existing datasets often suffer from limitations in scale, diversity, and quality, hindering the development and evaluation of IT models. To address this issue, we introduce MIT-10M, a large-scale parallel corpus of multilingual image translation with over 10M image-text pairs derived from real-world data, which has undergone extensive data cleaning and multilingual translation validation. It contains 840K images in three sizes, 28 categories, tasks with three levels of difficulty and 14 languages image-text pairs, which is a considerable improvement on existing datasets. We conduct extensive experiments to evaluate and train models on MIT-10M. The experimental results clearly indicate that our dataset has higher adaptability when it comes to evaluating the performance of the models in tackling challenging and complex image translation tasks in the real world. Moreover, the performance of the model fine-tuned with MIT-10M has tripled compared to the baseline model, further confirming its superiority.

Paper Structure

This paper contains 39 sections, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. IT Models
  4. Cascade Methods.
  5. End-to-End Methods.
  6. IT Datasets
  7. MIT-10M Construction
  8. Data collection
  9. Web Crawling.
  10. Image collection.
  11. Filtering and cleaning data.
  12. Data annotation
  13. Initial text recognition.
  14. Precise text recognition.
  15. Filtering sensitive content and text cleaning.
  16. ...and 24 more sections

Figures (8)

  • Figure 1: Categories and languages of MIT-10M. It includes 28 categories and 14 languages image-text pairs (8 languages images).
  • Figure 2: Examples of MIT-10M dataset. Each image contains the original text and the corresponding language. Additionally, we annotate the image category, the token length of the text and the number of bounding boxes and used them for the difficulty level. In addition to the original text, 13 languages translations were annotated.
  • Figure 3: Overview of MIT-10M dataset construction pipeline.
  • Figure 4: Distribution of the number of bounding boxes in the images and the token lengths in English text.
  • Figure 5: Example of images with different resolutions.
  • ...and 3 more figures