Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Annotation-Efficient Vision-Language Model Adaptation to the Polish Language Using the LLaVA Framework

Grzegorz Statkiewicz, Alicja Dobrzeniecka, Karolina Seweryn, Aleksandra Krasnodębska, Karolina Piosek, Katarzyna Bogusz, Sebastian Cygert, Wojciech Kusa

TL;DR

This work reproduces and adapts the LLaVA-Next methodology to create a set of Polish VLMs, relying on a fully automated pipeline for translating and filtering existing multimodal datasets, and complement this with synthetic Polish data for OCR and culturally specific tasks.

Abstract

Most vision-language models (VLMs) are trained on English-centric data, limiting their performance in other languages and cultural contexts. This restricts their usability for non-English-speaking users and hinders the development of multimodal systems that reflect diverse linguistic and cultural realities. In this work, we reproduce and adapt the LLaVA-Next methodology to create a set of Polish VLMs. We rely on a fully automated pipeline for translating and filtering existing multimodal datasets, and complement this with synthetic Polish data for OCR and culturally specific tasks. Despite relying almost entirely on automatic translation and minimal manual intervention to the training data, our approach yields strong results: we observe a +9.5% improvement over LLaVA-1.6-Vicuna-13B on a Polish-adapted MMBench, along with higher-quality captions in generative evaluations, as measured by human annotators in terms of linguistic correctness. These findings highlight that large-scale automated translation, combined with lightweight filtering, can effectively bootstrap high-quality multimodal models for low-resource languages. Some challenges remain, particularly in cultural coverage and evaluation. To facilitate further research, we make our models and evaluation dataset publicly available.

Annotation-Efficient Vision-Language Model Adaptation to the Polish Language Using the LLaVA Framework

TL;DR

This work reproduces and adapts the LLaVA-Next methodology to create a set of Polish VLMs, relying on a fully automated pipeline for translating and filtering existing multimodal datasets, and complement this with synthetic Polish data for OCR and culturally specific tasks.

Abstract

Most vision-language models (VLMs) are trained on English-centric data, limiting their performance in other languages and cultural contexts. This restricts their usability for non-English-speaking users and hinders the development of multimodal systems that reflect diverse linguistic and cultural realities. In this work, we reproduce and adapt the LLaVA-Next methodology to create a set of Polish VLMs. We rely on a fully automated pipeline for translating and filtering existing multimodal datasets, and complement this with synthetic Polish data for OCR and culturally specific tasks. Despite relying almost entirely on automatic translation and minimal manual intervention to the training data, our approach yields strong results: we observe a +9.5% improvement over LLaVA-1.6-Vicuna-13B on a Polish-adapted MMBench, along with higher-quality captions in generative evaluations, as measured by human annotators in terms of linguistic correctness. These findings highlight that large-scale automated translation, combined with lightweight filtering, can effectively bootstrap high-quality multimodal models for low-resource languages. Some challenges remain, particularly in cultural coverage and evaluation. To facilitate further research, we make our models and evaluation dataset publicly available.
Paper Structure (30 sections, 30 figures, 8 tables)

This paper contains 30 sections, 30 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-language models
  4. Visual instruction following datasets
  5. Polish large language models
  6. Model
  7. Data
  8. VLM-training datasets
  9. Polish language adaptation
  10. Pre-training data
  11. Instruction data
  12. General
  13. OCR
  14. Knowledge
  15. Counting
  16. ...and 15 more sections

Figures (30)

  • Figure 1: Comparative analysis of sample VLM predictions on two example images from our internal evaluation dataset. For each image, the human-provided prompt is shown, followed by our model and other baseline models' predictions. All predictions are presented in Appendix \ref{['app:internal']}.
  • Figure 2: Our custom dataset construction process.
  • Figure 3: Automatic evaluation on a subset of the XM3600 dataset for the content description quality criterion, using Claude Sonnet 4.5 as the judge.
  • Figure 4: Automatic evaluation on a subset of the XM3600 dataset for the linguistic correctness criterion, using Claude Sonnet 4.5 as the judge.
  • Figure 5: Human evaluation of image description quality on a subset of the XM3600 dataset.
  • ...and 25 more figures