Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Behind Maya: Building a Multilingual Vision Language Model

Nahid Alam, Karthik Reddy Kanjula, Surya Guthikonda, Timothy Chung, Bala Krishna S Vegesna, Abhipsha Das, Anthony Susevski, Ryan Sze-Yin Chan, S M Iftekhar Uddin, Shayekh Bin Islam, Roshan Santhosh, Snegha A, Drishti Sharma, Chen Liu, Isha Chaturvedi, Genta Indra Winata, Ashvanth. S, Snehanshu Mukherjee, Alham Fikri Aji

TL;DR

The paper tackles the gap in vision-language models for low-resource languages and cultural contexts by presenting Maya, an open-source multilingual VLM. It builds a multilingual pretraining dataset by expanding the LLaVA English corpus from 550K samples to 4.4M across eight languages using cascaded translation, back-translation, and human review, plus a prompt-engineering pipeline. Maya combines a SigLIP vision encoder with a multilingual Aya-23 8B LLM and a 2-layer projection, trained through pretraining on translated data and instruction-tuning on PALO to achieve strong multilingual performance. The authors provide an open-source pipeline and datasets, enabling broader multilingual multimodal research and practical deployment across diverse languages.

Abstract

In recent times, we have seen a rapid development of large Vision-Language Models (VLMs). They have shown impressive results on academic benchmarks, primarily in widely spoken languages but lack performance on low-resource languages and varied cultural contexts. To address these limitations, we introduce Maya, an open-source Multilingual VLM. Our contributions are: 1) a multilingual image-text pretraining dataset in eight languages, based on the LLaVA pretraining dataset; and 2) a multilingual image-text model supporting these languages, enhancing cultural and linguistic comprehension in vision-language tasks. Code available at https://github.com/nahidalam/maya.

Behind Maya: Building a Multilingual Vision Language Model

TL;DR

The paper tackles the gap in vision-language models for low-resource languages and cultural contexts by presenting Maya, an open-source multilingual VLM. It builds a multilingual pretraining dataset by expanding the LLaVA English corpus from 550K samples to 4.4M across eight languages using cascaded translation, back-translation, and human review, plus a prompt-engineering pipeline. Maya combines a SigLIP vision encoder with a multilingual Aya-23 8B LLM and a 2-layer projection, trained through pretraining on translated data and instruction-tuning on PALO to achieve strong multilingual performance. The authors provide an open-source pipeline and datasets, enabling broader multilingual multimodal research and practical deployment across diverse languages.

Abstract

In recent times, we have seen a rapid development of large Vision-Language Models (VLMs). They have shown impressive results on academic benchmarks, primarily in widely spoken languages but lack performance on low-resource languages and varied cultural contexts. To address these limitations, we introduce Maya, an open-source Multilingual VLM. Our contributions are: 1) a multilingual image-text pretraining dataset in eight languages, based on the LLaVA pretraining dataset; and 2) a multilingual image-text model supporting these languages, enhancing cultural and linguistic comprehension in vision-language tasks. Code available at https://github.com/nahidalam/maya.
Paper Structure (16 sections, 1 equation, 5 figures, 2 tables)

This paper contains 16 sections, 1 equation, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Dataset Creation and Filtering
  3. Dataset Creation Methodology
  4. Multilingual LLaVA Pretrain Dataset
  5. Prompt Engineering and Evaluation
  6. Translation Framework Design
  7. Scalable Dataset Generation
  8. Multilingual Multimodal Instruction Tuning
  9. Model Architecture
  10. Pretraining
  11. Finetuning
  12. Results
  13. Conclusion
  14. Appendix
  15. Author Affiliations
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Pretrain Dataset Preparation Process
  • Figure 2: Radar chart of N-gram averages by preamble
  • Figure 3: Maya Architecture adapted from LLaVA liu2024llavanext
  • Figure 4: Example image from LLaVA-Bench (In-the-Wild) liu2023llava.
  • Figure 5: Maya output for prompt (with image from Figure \ref{['fig:asianfood']}): Please describe the food in {language} in 1 sentence.