TowerVision: Understanding and Improving Multilinguality in Vision-Language Models
André G. Viveiros, Patrick Fernandes, Saul Santos, Sonal Sannigrahi, Emmanouil Zaranis, Nuno M. Guerreiro, Amin Farajian, Pierre Colombo, Graham Neubig, André F. T. Martins
TL;DR
TowerVision tackles the multilingual limitations of vision-language models by building open, end-to-end multilingual VLMs (image-text and video-text) on a multilingual text backbone (Tower+) and a multilingual vision encoder (SigLIP2). It introduces VisionBlocks, a curated data suite combining translated, synthetic, and text-only data to strengthen cross-lingual grounding, and executes a three-stage training pipeline that includes projector pretraining, vision finetuning on VisionBlocks, and video finetuning on multilingual video data. The work demonstrates that expanding multilingual training data and using robust multilingual backbones yields strong cross-lingual generalization, with TowerVision achieving competitive to state-of-the-art results on vision-language benchmarks and state-of-the-art performance on Multi30K, while TowerVideo shows benefits in culturally-aware multilingual video tasks. The authors release all models, data, and training recipes, providing a practical path toward accessible, culturally aware multilingual VLMs, and highlight that multilingual supervision during alignment may offer diminishing returns relative to data quality and backbone strength, informing future open multilingual VLM development.
Abstract
Despite significant advances in vision-language models (VLMs), most existing work follows an English-centric design process, limiting their effectiveness in multilingual settings. In this work, we provide a comprehensive empirical study analyzing the impact of several multilingual design choices, such as training data composition, encoder selection, and text backbones. The result is TowerVision, a family of open multilingual VLMs for both image-text and video-text tasks, built upon the multilingual text-only model Tower+. TowerVision achieves competitive performance on multiple multimodal multilingual benchmarks and shows particular strength in culturally grounded tasks and multimodal translation. By incorporating visual and cultural context during fine-tuning, our models surpass existing approaches trained on substantially larger datasets, as demonstrated on ALM-Bench and Multi30K (image tasks) and ViMUL-Bench (video tasks). Alongside the models, we release VisionBlocks, a high-quality, curated vision-language dataset. Our findings highlight that multilingual vision-language training data substantially improves cross-lingual generalization -- both from high-resource to underrepresented languages and vice versa -- and that instruction-tuned LLMs are not always the optimal initialization point. To support further research, we publicly release all models, data, and training recipes.