Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages

TL;DR

Pangea tackles the lack of culturally inclusive multilingual multimodal models by building PangeaIns, a 6M-instruction, 39-language dataset, and PangeaBench, a comprehensive evaluation suite over 47 languages. The authors train Pangea-7B on a two-stage vision-language pipeline (LLaVA-Next backbone) and demonstrate that open-source multilingual models can surpass prior open baselines and approach some proprietary systems on multilingual tasks. Ablation studies show the critical roles of English data proportion, language prevalence, and the number of multimodal training samples in shaping performance. The work emphasizes openness by fully releasing data, code, and trained checkpoints to promote accessible, culturally aware MLLMs.

Abstract

Despite recent advances in multimodal large language models (MLLMs), their development has predominantly focused on English- and western-centric datasets and tasks, leaving most of the world's languages and diverse cultural contexts underrepresented. This paper introduces Pangea, a multilingual multimodal LLM trained on PangeaIns, a diverse 6M instruction dataset spanning 39 languages. PangeaIns features: 1) high-quality English instructions, 2) carefully machine-translated instructions, and 3) culturally relevant multimodal tasks to ensure cross-cultural coverage. To rigorously assess models' capabilities, we introduce PangeaBench, a holistic evaluation suite encompassing 14 datasets covering 47 languages. Results show that Pangea significantly outperforms existing open-source models in multilingual settings and diverse cultural contexts. Ablation studies further reveal the importance of English data proportions, language popularity, and the number of multimodal training samples on overall performance. We fully open-source our data, code, and trained checkpoints, to facilitate the development of inclusive and robust multilingual MLLMs, promoting equity and accessibility across a broader linguistic and cultural spectrum.

Figures (27)

• Figure 1: Overview of the aggregate performance of various multimodal LLMs on PangeaBench. Our Pangea-7B demonstrates comparable performance to SoTA open-source models in English settings, while significantly outperforming them in multilingual scenarios.
• Figure 2: Statistics of PangeaIns, comprising 6M multimodal instructions in 39 languages. The distribution of multilingual training data shows the percent of instances for each language among the multilingual instructions. PangeaIns includes general instructions, document and chart question answering, captioning, domain-specific, culturally relevant, and text-only instructions.
• Figure 3: Overview of multicultural understanding instructions data generation pipeline.
• Figure 4: Overview of PangeaBench, which contains 5 multimodal and 3 text tasks covering 14 datasets (including two newly curated xChatBench and xMMMU datasets). The table provides details about the datasets, while the figure shows evaluation examples from five different multimodal eval tasks in our PangeaBench.
• Figure 5: Scaling effect of training samples on English and multilingual scores across datasets.