Rethinking Overlooked Aspects in Vision-Language Models

TL;DR

The paper argues that current vision-language model progress hinges on data efficiency rather than sheer scale. It systematically analyzes pre-training data and instruction-tuning dataset selection, introducing Individual Select to identify efficient SFT datasets and demonstrating that naive expansion of pre-training data can degrade performance. Using LLaVA-1.5 as a simple baseline, it establishes a stable, multi-benchmark framework and shows that curated SFT data yield substantial gains over indiscriminate fine-tuning. The work offers a practical, data-centric roadmap for future LVLM development, emphasizing data quality, diversity, and targeted instruction tuning to improve generalization across diverse tasks.

Abstract

Recent advancements in large vision-language models (LVLMs), such as GPT4-V and LLaVA, have been substantial. LLaVA's modular architecture, in particular, offers a blend of simplicity and efficiency. Recent works mainly focus on introducing more pre-training and instruction tuning data to improve model's performance. This paper delves into the often-neglected aspects of data efficiency during pre-training and the selection process for instruction tuning datasets. Our research indicates that merely increasing the size of pre-training data does not guarantee improved performance and may, in fact, lead to its degradation. Furthermore, we have established a pipeline to pinpoint the most efficient instruction tuning (SFT) dataset, implying that not all SFT data utilized in existing studies are necessary. The primary objective of this paper is not to introduce a state-of-the-art model, but rather to serve as a roadmap for future research, aiming to optimize data usage during pre-training and fine-tuning processes to enhance the performance of vision-language models.

Figures (4)

• Figure 1: Performance of LLaVA-1.5 using LLaVA-1.5-665K and our SFT dataset. We have developed a strategy, termed Individual Select, which is designed to select the most effective datasets from a plethora of publicly available SFT datasets. The LLaVA-1.5 model, fine-tuned with Vicuna-7Bvicuna2023 on the final composition of SFT datasets that we have obtained, yields substantial improvements compared to the baseline. The original MME scores are mapped to a range of 0 to 100.
• Figure 2: Pre-training data scaling law. We investigated this phenomenon in large vision-language models using three different types of mainstream Large Language Models (LLMs). As we increased the size of the pre-training dataset from 1 million to 100 million samples, the model's performance remained nearly consistent, with some instances of degradation observed.
• Figure 3: The workflow of Individual Select (a) and the SFT datasets we finally choose to be added to the baseline dataset.
• Figure 4: We observe consistent improvements as we incorporate these selected datasets from each category. The original MME scores are mapped to a range of 0 to 100. Average: the average score of the four metrics.