Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Maya: An Instruction Finetuned Multilingual Multimodal Model

Nahid Alam, Karthik Reddy Kanjula, Surya Guthikonda, Timothy Chung, Bala Krishna S Vegesna, Abhipsha Das, Anthony Susevski, Ryan Sze-Yin Chan, S M Iftekhar Uddin, Shayekh Bin Islam, Roshan Santhosh, Snegha A, Drishti Sharma, Chen Liu, Isha Chaturvedi, Genta Indra Winata, Ashvanth. S, Snehanshu Mukherjee, Alham Fikri Aji

TL;DR

Maya targets the core gap in vision-language models: robust multilingual and multicultural understanding with safety guarantees. The authors introduce a toxicity-filtered, eight-language pretraining dataset built on LLaVA, paired with a Maya mVLM that uses a SigLIP vision encoder and an Aya-23 8B LLM, followed by instruction-tuning with PALO data. They demonstrate competitive multilingual performance against open-source baselines and provide a toxicity-free variant to study safety impacts, while releasing the codebase for open collaboration. This work advances accessible, culturally aware multimodal reasoning in low-resource languages and establishes a foundation for further multilingual and safety-focused VLM research.

Abstract

The rapid development of large Vision-Language Models (VLMs) has led to impressive results on academic benchmarks, primarily in widely spoken languages. However, significant gaps remain in the ability of current VLMs to handle low-resource languages and varied cultural contexts, largely due to a lack of high-quality, diverse, and safety-vetted data. Consequently, these models often struggle to understand low-resource languages and cultural nuances in a manner free from toxicity. To address these limitations, we introduce Maya, an open-source Multimodal Multilingual model. Our contributions are threefold: 1) a multilingual image-text pretraining dataset in eight languages, based on the LLaVA pretraining dataset; 2) a thorough analysis of toxicity within the LLaVA dataset, followed by the creation of a novel toxicity-free version across eight languages; and 3) a multilingual image-text model supporting these languages, enhancing cultural and linguistic comprehension in vision-language tasks. Code available at https://github.com/nahidalam/maya.

Maya: An Instruction Finetuned Multilingual Multimodal Model

TL;DR

Maya targets the core gap in vision-language models: robust multilingual and multicultural understanding with safety guarantees. The authors introduce a toxicity-filtered, eight-language pretraining dataset built on LLaVA, paired with a Maya mVLM that uses a SigLIP vision encoder and an Aya-23 8B LLM, followed by instruction-tuning with PALO data. They demonstrate competitive multilingual performance against open-source baselines and provide a toxicity-free variant to study safety impacts, while releasing the codebase for open collaboration. This work advances accessible, culturally aware multimodal reasoning in low-resource languages and establishes a foundation for further multilingual and safety-focused VLM research.

Abstract

The rapid development of large Vision-Language Models (VLMs) has led to impressive results on academic benchmarks, primarily in widely spoken languages. However, significant gaps remain in the ability of current VLMs to handle low-resource languages and varied cultural contexts, largely due to a lack of high-quality, diverse, and safety-vetted data. Consequently, these models often struggle to understand low-resource languages and cultural nuances in a manner free from toxicity. To address these limitations, we introduce Maya, an open-source Multimodal Multilingual model. Our contributions are threefold: 1) a multilingual image-text pretraining dataset in eight languages, based on the LLaVA pretraining dataset; 2) a thorough analysis of toxicity within the LLaVA dataset, followed by the creation of a novel toxicity-free version across eight languages; and 3) a multilingual image-text model supporting these languages, enhancing cultural and linguistic comprehension in vision-language tasks. Code available at https://github.com/nahidalam/maya.

Paper Structure

This paper contains 20 sections, 1 equation, 15 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multilingual Large Language Models
  4. Multimodal Large Language Models
  5. Multimodal Multilingual Datasets
  6. Evaluating Multimodal Models
  7. Dataset Creation and Filtering
  8. Dataset Creation Methodology
  9. Initial Dataset Processing
  10. Prompt Engineering and Evaluation
  11. Translation Framework Design
  12. Scalable Dataset Generation
  13. Dataset Toxicity Filtering
  14. Multilingual Multimodal Instruction Tuning
  15. Model Architecture
  16. ...and 5 more sections

Figures (15)

  • Figure 1: Pretrain Dataset Preparation Process
  • Figure 2: N-gram values by language and preamble type. These values are average of 1-gram, 2-gram, 3-gram and 4-gram
  • Figure 3: Radar chart of N-gram averages by preamble
  • Figure 4: Image Toxicity Analysis with LLaVAGuard
  • Figure 5: Image Caption Toxicity Analysis with Toxic-BERT
  • ...and 10 more figures