ICONS: Influence Consensus for Vision-Language Data Selection

TL;DR

ICONS tackles redundancy in large vision-language instruction-tuning data by estimating per-sample influence on validation performance across multiple tasks and then aggregating these signals through a majority-voting consensus. The method employs a two-stage process: a specialist stage computes task-specific influence scores via first-order gradients with a LoRA-warmed-up model, and a generalist stage converts these scores into cross-task votes to select a compact subset (e.g., 20%) that preserves or even exceeds full-dataset performance. Empirically, ICONS achieves near-full performance on LLaVA-665K and strong results on Cambrian-7M and Vision-Flan-186K, with 60% selection sometimes surpassing full-dataset training. The approach generalizes to unseen tasks and architectures and is complemented by releases of compact data subsets, enabling resource-efficient development of vision-language models while reducing computation and storage costs.

Abstract

Training vision-language models via instruction tuning relies on large data mixtures spanning diverse tasks and domains, yet these mixtures frequently include redundant information that increases computational costs without proportional gains. Existing methods typically rely on task-agnostic heuristics to estimate data importance, limiting their effectiveness across tasks. We introduce ICONS, a gradient-based Influence CONsensus approach for vision-language data Selection. Our method leverages first-order training dynamics to estimate each example's influence on validation performance, then aggregates these estimates across tasks via majority voting. This cross-task consensus identifies consistently valuable data points while mitigating score calibration and outlier sensitivity, enabling robust and scalable data selection for diverse multitask mixtures. Models trained on our selected 20% data subset from LLAVA-665K (respectively: from CAMBRIAN-7M, from VISION-FLAN-186K) retain 98.6% (respectively: 98.8%, 99.8%) of full-dataset performance. We demonstrate that our selected data generalizes to unseen tasks and model architectures, and release three compact subsets LLAVA-ICONS-133K, CAMBRIAN-ICONS-1.4M, and VISION-FLAN-ICONS-37K for efficient vision-language model development.

Figures (22)

• Figure 1: Influence consensus for vision-language data selection. (Left) Given a large-scale visual instruction tuning dataset (LLaVA-665K), our method uses majority voting across task-specific influence scores to identify training samples that are consistently influential across multiple tasks, forming a compact 20% subset (LLaVA-ICONS-133K) with data points achieving influence consensus. (Right) The radar plot compares performance between LLaVA-665K and our selected subset, showing the selected subset achieves comparable results to the full dataset.
• Figure 2: ICONS. The Specialist stage (left) processes each task through: (1) warmup training on a small subset, (2) gradient computation for training and validation data, and (3) influence matrix computation for per-task scores. The Generalist stage (right) performs Influence Consensus to aggregate information across tasks. For each task, samples scoring above a percentile threshold (e.g., $80^{\text{th}}$ for 20% selection) receive a vote. Votes are summed across tasks, and samples with the highest vote counts are selected, creating a compact yet effective training dataset that performs well across all tasks.
• Figure 3: Unseen task generalization across methods. Performance on eight unseen benchmarks using 20% selected subsets, sorted by the average performance across 5 random selection runs (left: lower, right: higher). Shaded region shows mean and standard deviation across random runs. Our ICONS consistently outperforms all baselines. Full detailed results are in Appendix Tab. \ref{['tab:cross_task']}.
• Figure 4: Different selection ratios.ICONS consistently outperforms all baselines across different selection ratios and remarkably exceeds 102% at a 60% selection ratio.
• Figure 5: Ablation studies on (left) projection dimension and (right) warm-up ratio.