Constructing Multimodal Datasets from Scratch for Rapid Development of a Japanese Visual Language Model

TL;DR

This work takes Japanese as a non-English language and proposes a method for rapidly creating Japanese multimodal datasets from scratch, and shows that a VLM trained on these native datasets outperforms those relying on machine-translated content.

Abstract

To develop high-performing Visual Language Models (VLMs), it is essential to prepare multimodal resources, such as image-text pairs, interleaved data, and instruction data. While multimodal resources for English are abundant, there is a significant lack of corresponding resources for non-English languages, such as Japanese. To address this problem, we take Japanese as a non-English language and propose a method for rapidly creating Japanese multimodal datasets from scratch. We collect Japanese image-text pairs and interleaved data from web archives and generate Japanese instruction data directly from images using an existing VLM. Our experimental results show that a VLM trained on these native datasets outperforms those relying on machine-translated content.

Figures (7)

• Figure 1: We propose VILA-jp, a novel Japanese vision and language model. For each step of pre-training and instruction tuning, we construct tailored million-scale image-text dataset ($\blacksquare$) from interleaved data.
• Figure 2: Examples of text generated by each model in response to a question from the Japanese-Heron-Bench. Green indicates the correct word and red indicates the wrong word.
• Figure 3: Example of how ROUGE-L scores can be misleading when evaluating Japanese VQA responses. Three different LLM-generated answers are shown alongside their corresponding ROUGE-L scores.
• Figure 4: Additional qualitative examples in the Heron Bench.
• Figure 5: Continuous qualitative examples in the Heron Bench.