M5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks

TL;DR

The paper presents M5, a diverse, multimodal benchmark designed to evaluate Large Multimodal Models across multilingual and multicultural vision-language tasks. It introduces eight datasets spanning five tasks and 41 languages, plus two novel datasets (M5-VGR and M5-VLOD) that emphasize underrepresented languages and cross-cultural content, including a new Visio-Linguistic Outlier Detection task. Through extensive evaluations of eighteen recent LMMs, the work reveals substantial language-resource disparities and shows that larger models do not universally outperform smaller ones in multilingual settings. It underscores the need for diverse training data and robust architectures to achieve globally equitable vision-language AI and provides public code and data to catalyze further research. Overall, M5 highlights critical gaps and provides a scalable framework to benchmark and improve multilingual and multicultural visio-linguistic capabilities.

Abstract

Since the release of ChatGPT, the field of Natural Language Processing has experienced rapid advancements, particularly in Large Language Models (LLMs) and their multimodal counterparts, Large Multimodal Models (LMMs). Despite their impressive capabilities, LLMs often exhibit significant performance disparities across different languages and cultural contexts, as demonstrated by various text-only benchmarks. However, current research lacks such benchmarks for multimodal visio-linguistic settings. This work fills this gap by introducing M5, the first comprehensive benchmark designed to evaluate LMMs on diverse vision-language tasks within a multilingual and multicultural context. M5 includes eight datasets covering five tasks and $41$ languages, with a focus on underrepresented languages and culturally diverse images. Furthermore, we introduce two novel datasets, M5-VGR and M5-VLOD, including a new Visio-Linguistic Outlier Detection task, in which all evaluated open-source models fail to significantly surpass the random baseline. Through extensive evaluation and analyses, we highlight substantial task-agnostic performance disparities between high- and low-resource languages. Moreover, we show that larger models do not necessarily outperform smaller ones in a multilingual setting.

Figures (39)

• Figure 1: An overview of the average performance of the models on the datasets included in the M5 benchmark. For xFlickrCO and XM3600, we report BERTScore F1. For the other datasets, the accuracy metric is reported.
• Figure 2: An informative overview of the M5 Benchmark introduced in this work.
• Figure 3: An Zulu example of the novel M5-VGR dataset. Hypothesis:"Isithombe sokuqala nesithombe sesibili sibonisa iqanda elisehhokweni. (The first picture and the second picture show the egg on the head.)", Label:False
• Figure 4: A Swahili example of the novel M5-VLOD dataset. Hypothesis: "Picha zote zinaonyesha sabuni inayotumika kwa mikono na mwili bila mtu yeyote. (All the images show soap applied to the hands and body without anyone.)", Outlier:$1$.
• Figure 5: Models' performances on all datasets aggregated by language or language taxonomy classes.