mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus

TL;DR

mOSCAR introduces a first-of-its-kind large-scale multilingual and multimodal document corpus harvested from the web, spanning 163 languages with 303M documents and 1.15B images. The dataset undergoes extensive, multi-layer filtering for text quality, safety, and cross-modal relevance, including PII and CSAM safeguards. The authors train multilingual OpenFlamingo models on mOSCAR (with and without caption data) and demonstrate notable few-shot and zero-shot gains across diverse multilingual image-text tasks, including VQA, translation, and captioning, outperforming caption-only baselines and some multilingual peers. This work highlights the value of truly interleaved, document-level multimodal data for expanding language coverage in mLLMs, while acknowledging biases and safety considerations inherent in web-crawled corpora. The public CC BY 4.0 release aims to accelerate multilingual multimodal research and applications across many languages and cultures.

Abstract

Multimodal Large Language Models (mLLMs) are trained on a large amount of text-image data. While most mLLMs are trained on caption-like data only, Alayrac et al. (2022) showed that additionally training them on interleaved sequences of text and images can lead to the emergence of in-context learning capabilities. However, the dataset they used, M3W, is not public and is only in English. There have been attempts to reproduce their results but the released datasets are English-only. In contrast, current multilingual and multimodal datasets are either composed of caption-like only or medium-scale or fully private data. This limits mLLM research for the 7,000 other languages spoken in the world. We therefore introduce mOSCAR, to the best of our knowledge the first large-scale multilingual and multimodal document corpus crawled from the web. It covers 163 languages, 303M documents, 200B tokens and 1.15B images. We carefully conduct a set of filtering and evaluation steps to make sure mOSCAR is sufficiently safe, diverse and of good quality. We additionally train two types of multilingual model to prove the benefits of mOSCAR: (1) a model trained on a subset of mOSCAR and captioning data and (2) a model trained on captioning data only. The model additionally trained on mOSCAR shows a strong boost in few-shot learning performance across various multilingual image-text tasks and benchmarks, confirming previous findings for English-only mLLMs. The dataset is released under the Creative Commons CC BY 4.0 license and can be accessed here: https://huggingface.co/datasets/oscar-corpus/mOSCAR

Figures (10)

• Figure 1: Example of a French document from mOSCAR.
• Figure 2: Distributions of numbers of tokens and images per document.
• Figure 3: Perplexity of 100K random documents from different datasets.
• Figure 4: Score differences averaged over benchmarks and languages between the model trained on mOSCAR + text-image pairs and the model trained only on text-image pairs. Bold is best result.
• Figure 5: Example of a French document.