Aya Vision: Advancing the Frontier of Multilingual Multimodality

TL;DR

Aya Vision tackles the core challenges of multilingual multimodal AI by introducing a scalable synthetic multilingual data framework and a training-free cross-modal merging mechanism. The data framework combines distillation-based recaptioning, robust filtering, and a hybrid translation-and-rephrasing pipeline to produce high-quality instruction data across 23 languages. The authors further mitigate catastrophic forgetting with cross-modal merging, achieving strong multimodal results while preserving text-only capabilities, demonstrated by state-of-the-art performance for Aya-Vision-8B and competitive results for Aya-Vision-32B against models many times larger. The work is anchored by AyaVisionBench and m-WildVision, offering a comprehensive suite for real-world multilingual multimodal evaluation and establishing a practical, compute-efficient path to high performance in diverse languages and modalities.

Abstract

Building multimodal language models is fundamentally challenging: it requires aligning vision and language modalities, curating high-quality instruction data, and avoiding the degradation of existing text-only capabilities once vision is introduced. These difficulties are further magnified in the multilingual setting, where the need for multimodal data in different languages exacerbates existing data scarcity, machine translation often distorts meaning, and catastrophic forgetting is more pronounced. To address the aforementioned challenges, we introduce novel techniques spanning both data and modeling. First, we develop a synthetic annotation framework that curates high-quality, diverse multilingual multimodal instruction data, enabling Aya Vision models to produce natural, human-preferred responses to multimodal inputs across many languages. Complementing this, we propose a cross-modal model merging technique that mitigates catastrophic forgetting, effectively preserving text-only capabilities while simultaneously enhancing multimodal generative performance. Aya-Vision-8B achieves best-in-class performance compared to strong multimodal models such as Qwen-2.5-VL-7B, Pixtral-12B, and even much larger Llama-3.2-90B-Vision. We further scale this approach with Aya-Vision-32B, which outperforms models more than twice its size, such as Molmo-72B and LLaMA-3.2-90B-Vision. Our work advances multilingual progress on the multi-modal frontier, and provides insights into techniques that effectively bend the need for compute while delivering extremely high performance.

Figures (25)

• Figure 1: Aya Vision models achieve state-of-the-art multilingual performance across both multimodal and text-only tasks. We report multimodal and text-only win rates against Pangea-7B yue2024pangea, averaged over 23 languages. Aya-Vision-8B achieves best-in-class multimodal performance without compromising text capabilities, while Aya-Vision-32B outperforms all baselines, including much larger models such as Llama-3.2-90B-Vision grattafiori2024llama, establishing an optimal balance between efficiency and cross-modal strength.
• Figure 2: Aya Vision establishes a new Pareto frontier in the performance-efficiency trade-off. We show multimodal win rates against Pangea-7B, with respect to the number of parameters for each model.
• Figure 3: Our synthetic annotation pipeline enables diverse, high quality responses for multimodal instructions. The pipeline consists of three core stages: (1) distillation-based recaptioning, (2) machine translation, and (3) rephrasing. We highlight common machine translation errors, such as unknown tokens (e.g. consistency, lit candle) or mistranslations, as in the case of 'French press' rendered as 'French media' due to lexical ambiguity in the word 'press'. Rephrasing helps to resolve such issues, improving both the fluency and semantic accuracy of translations.
• Figure 4: Overview of our multilingual multimodal SFT mixture from various task categories. Left: Number of samples across data sources and tasks categories used in training. Right: Visual breakdown of dataset source distributions.
• Figure 5: Degradation in text-only win-rates after multimodal training. Each model is compared to their initial LLM on mArenaHard dang2024aya. We see that only including a percentage of text-only data in the final multimodal training mix is insufficient to retain open-ended generative performance.