Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation

Israfel Salazar, Manuel Fernández Burda, Shayekh Bin Islam, Arshia Soltani Moakhar, Shivalika Singh, Fabian Farestam, Angelika Romanou, Danylo Boiko, Dipika Khullar, Mike Zhang, Dominik Krzemiński, Jekaterina Novikova, Luísa Shimabucoro, Joseph Marvin Imperial, Rishabh Maheshwary, Sharad Duwal, Alfonso Amayuelas, Swati Rajwal, Jebish Purbey, Ahmed Ruby, Nicholas Popovič, Marek Suppa, Azmine Toushik Wasi, Ram Mohan Rao Kadiyala, Olga Tsymboi, Maksim Kostritsya, Bardia Soltani Moakhar, Gabriel da Costa Merlin, Otávio Ferracioli Coletti, Maral Jabbari Shiviari, MohammadAmin farahani fard, Silvia Fernandez, María Grandury, Dmitry Abulkhanov, Drishti Sharma, Andre Guarnier De Mitri, Leticia Bossatto Marchezi, Setayesh Heydari, Johan Obando-Ceron, Nazar Kohut, Beyza Ermis, Desmond Elliott, Enzo Ferrante, Sara Hooker, Marzieh Fadaee

TL;DR

Kaleidoscope addresses a critical gap in vision-language evaluation by delivering the largest in-language, multimodal exam benchmark across 18 languages and 14 subjects. It combines a rigorous, globally-coordinated data collection process with an MCQA framework to assess cross-linguistic and visual reasoning, supported by both open-weight and closed models. Key findings show strong modality gaps, pronounced language-resource biases, and STEM-specific deficiencies, with model size and textual augmentations offering nuanced but limited gains. The benchmark advances culturally authentic evaluation and calls for more inclusive datasets and evaluation paradigms to ensure reliable, globally relevant AI systems.

Abstract

The evaluation of vision-language models (VLMs) has mainly relied on English-language benchmarks, leaving significant gaps in both multilingual and multicultural coverage. While multilingual benchmarks have expanded, both in size and languages, many rely on translations of English datasets, failing to capture cultural nuances. In this work, we propose Kaleidoscope, as the most comprehensive exam benchmark to date for the multilingual evaluation of vision-language models. Kaleidoscope is a large-scale, in-language multimodal benchmark designed to evaluate VLMs across diverse languages and visual inputs. Kaleidoscope covers 18 languages and 14 different subjects, amounting to a total of 20,911 multiple-choice questions. Built through an open science collaboration with a diverse group of researchers worldwide, Kaleidoscope ensures linguistic and cultural authenticity. We evaluate top-performing multilingual vision-language models and find that they perform poorly on low-resource languages and in complex multimodal scenarios. Our results highlight the need for progress on culturally inclusive multimodal evaluation frameworks.

Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation

TL;DR

Kaleidoscope addresses a critical gap in vision-language evaluation by delivering the largest in-language, multimodal exam benchmark across 18 languages and 14 subjects. It combines a rigorous, globally-coordinated data collection process with an MCQA framework to assess cross-linguistic and visual reasoning, supported by both open-weight and closed models. Key findings show strong modality gaps, pronounced language-resource biases, and STEM-specific deficiencies, with model size and textual augmentations offering nuanced but limited gains. The benchmark advances culturally authentic evaluation and calls for more inclusive datasets and evaluation paradigms to ensure reliable, globally relevant AI systems.

Abstract

The evaluation of vision-language models (VLMs) has mainly relied on English-language benchmarks, leaving significant gaps in both multilingual and multicultural coverage. While multilingual benchmarks have expanded, both in size and languages, many rely on translations of English datasets, failing to capture cultural nuances. In this work, we propose Kaleidoscope, as the most comprehensive exam benchmark to date for the multilingual evaluation of vision-language models. Kaleidoscope is a large-scale, in-language multimodal benchmark designed to evaluate VLMs across diverse languages and visual inputs. Kaleidoscope covers 18 languages and 14 different subjects, amounting to a total of 20,911 multiple-choice questions. Built through an open science collaboration with a diverse group of researchers worldwide, Kaleidoscope ensures linguistic and cultural authenticity. We evaluate top-performing multilingual vision-language models and find that they perform poorly on low-resource languages and in complex multimodal scenarios. Our results highlight the need for progress on culturally inclusive multimodal evaluation frameworks.

Paper Structure

This paper contains 38 sections, 5 figures, 14 tables.

Table of Contents

  1. Introduction
  2. The Kaleidoscope Benchmark
  3. Global Collaboration
  4. Data Pipeline
  5. Data Statistics
  6. Experimental Setup
  7. Models
  8. Evaluation Setup
  9. Evaluation Metrics
  10. Results
  11. Overall Performance
  12. Not All Image Types are Equal
  13. Resource and Script Sensitivity in Models
  14. STEM Questions Expose Model Deficiencies
  15. Analysis
  16. ...and 23 more sections

Figures (5)

  • Figure 1: Overview of the Kaleidoscope Benchmark. (a) Multilingual-Multimodal MCQ Samples (b) Language and Multimodal Samples Distribution. (c) Exam Category Breakdown.
  • Figure 2: Model Performance Analysis on Kaleidoscope. (a) Accuracy (%) of models on multimodal and text-only questions, highlighting low performance on multimodal samples. (b) Accuracy (%) by script type, revealing biases for latin scripts. Accuracy over valid responses is used to generate both figures. Identity line is added to show parity.
  • Figure 3: Multimodal Accuracy by Language in Kaleidoscope. Reports performance (accuracy %) for closed models and open-weight models on multimodal questions.
  • Figure 4: Model Size Analysis for Qwen2.5-VL Models. Performance improvement across three model sizes (3B, 7B, 32B, and 72B parameters) on Kaleidoscope's multimodal tasks, demonstrating consistent gains from increased model capacity. Note that x-axis is shown in log-scale.
  • Figure 5: Distribution of the number of format errors for each model/language combination. The languages are represented in their ISO 639 (set 1) code.