VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning

TL;DR

VisNec (Visual Necessity Score) is proposed, a principled data selection framework that measures the marginal contribution of visual input during instruction tuning and provides an effective solution for both efficient and robust multimodal instruction tuning.

Abstract

The effectiveness of multimodal instruction tuning depends not only on dataset scale, but critically on whether training samples genuinely require visual reasoning. However, existing instruction datasets often contain a substantial portion of visually redundant samples (solvable from text alone), as well as multimodally misaligned supervision that can degrade learning. To address this, we propose VisNec (Visual Necessity Score), a principled data selection framework that measures the marginal contribution of visual input during instruction tuning. By comparing predictive loss with and without visual context, VisNec identifies whether a training instance is vision-critical, redundant, or misaligned. To preserve task diversity, we combine VisNec with semantic clustering and select high-necessity samples within each cluster. Across 10 downstream benchmarks, training on only 15% of the LLaVA-665K dataset selected by VisNec achieves 100.2% of full-data performance. On the smaller Vision-Flan-186K dataset, our selection not only further reduces data size but also surpasses full-data training by 15.8%. These results demonstrate that measuring and leveraging visual necessity provides an effective solution for both efficient and robust multimodal instruction tuning. Codes and selected subsets will be released upon acceptance.

Figures (4)

• Figure 1: Illustration of the VisNec score, which measures the contribution of visual input in multimodal instruction tuning. Samples are categorized as Misaligned (VisNec < 0, visual input degrades prediction), Redundant ($\approx 0$, visual input provides no additional benefit), or Vision-Critical (> 0, visual input substantially improves prediction).
• Figure 2: Overview of the VisNec Data Selection Framework. Each sample undergoes two forward passes to compute its VisNec score. Samples with non-positive scores are filtered out, and the remainder are grouped via K-Means clustering, from which the top-$r$% are selected within each cluster for fine-tuning.
• Figure 3: Qualitative comparison of Low VisNec ($-1.2087$) and High VisNec ($3.4727$) samples, illustrating misaligned, redundant and visual dependent samples.
• Figure 4: (a) Impact of the number of clusters $k$ in K-means. (b) Impact of the sampling ratio $\rho$.