Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection

TL;DR

The paper tackles data quality in vision-language instruction tuning by introducing Self-Filter, a method that lets the VLM itself act as a data filter. It trains a lightweight score net in a two-stage process to estimate sample difficulty from multimodal features, then selects the hardest, most diverse instructions to fine-tune the model, without relying on external evaluation tasks. Empirically, Self-Filter achieves state-of-the-art data efficiency on LLaVA and MiniGPT-4, matching or surpassing full-data baselines with only about 15% of the data and with demonstrable gains from the diversity penalty. The approach preserves generalization and offers a practical, evaluation-free pathway to high-quality instruction tuning for vision-language models.

Abstract

Data selection in instruction tuning emerges as a pivotal process for acquiring high-quality data and training instruction-following large language models (LLMs), but it is still a new and unexplored research area for vision-language models (VLMs). Existing data selection approaches on LLMs either rely on single unreliable scores, or use downstream tasks for selection, which is time-consuming and can lead to potential over-fitting on the chosen evaluation datasets. To address this challenge, we introduce a novel dataset selection method, Self-Filter, that utilizes the VLM itself as a filter. This approach is inspired by the observation that VLMs benefit from training with the most challenging instructions. Self-Filter operates in two stages. In the first stage, we devise a scoring network to evaluate the difficulty of training instructions, which is co-trained with the VLM. In the second stage, we use the trained score net to measure the difficulty of each instruction, select the most challenging samples, and penalize similar samples to encourage diversity. Comprehensive experiments on LLaVA and MiniGPT-4 show that Self-Filter can reach better results compared to full data settings with merely about 15% samples, and can achieve superior performance against competitive baselines.

Figures (3)

• Figure 1: Illustration of the stage 1 of Self-Filter. In the target vision-language models, samples that are more challenging usually yield higher losses. During the training process, the loss of each instruction is combined with a learnable weight generated by the score net. Through minimizing this weighted loss function $\mathcal{L}$, the instructions with higher loss tend to have lower weights. It is important to note that in stage 1, both the target VLM and the score net are actively trained, with the pre-trained feature extractors remaining frozen. Following this, stage 2 involves employing the score net to estimate the weights for each instruction, in which a lower weight signifies a higher difficulty level for the model.
• Figure 2: Main results on MiniGPT-4 zhu2023minigpt. w/o Diversity represents removing the diversity module in our method. Self-Filter achieves the best results on most tasks, and has a higher average accuracy.
• Figure 3: Experimental results under difference target pruning size $m$.