CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization

TL;DR

CoIDO tackles the heavy data and compute costs of visual instruction tuning for multimodal language models by jointly optimizing data importance and diversity with a lightweight plug-in scorer trained on only a small random subset. Using a homoscedastic uncertainty formulation, CoIDO couples the two objectives into a single loss that guides sample scoring, enabling efficient and scalable data selection. The method applies spectral clustering to capture diversity and uses a per-sample CoIDO score to select a balanced, informative subset; experiments on LLaVA-665K and LLaVA-1.5-7B-LoRA show that training with 20% of data achieves about 98.2% of full-data performance across ten tasks, while dramatically reducing training data cost and FLOPs. CoIDO also demonstrates strong generalizability and cross-domain transferability, suggesting practical impact for resource-constrained settings and broader multimodal fine-tuning applications.

Abstract

Multimodal large language models (MLLMs) rely heavily on instruction tuning to align vision and language capabilities, yet the computational cost of training on large-scale datasets remains a major bottleneck. Existing data selection methods aim to mitigate this by selecting important and diverse subsets, but they often suffer from two critical drawbacks: high computational overhead from processing the entire dataset and suboptimal data selection due to separate treatment of importance and diversity. We introduce CoIDO, a novel dual-objective framework that jointly optimizes data importance and diversity to overcome these challenges. Unlike existing approaches that require costly evaluations across the whole dataset, CoIDO employs a lightweight plug-in scorer. This scorer is trained on just a small random sample of data to learn the distribution of the candidate set, drastically reducing computational demands. By leveraging a homoscedastic uncertainty-based formulation, CoIDO effectively balances importance and diversity during training, enabling efficient and scalable data selection. In our experiments, we trained the CoIDO scorer using only 20 percent of randomly sampled data. Once trained, CoIDO was applied to the entire dataset to select a 20 percent subset for instruction tuning. On the widely used LLaVA-1.5-7B model across ten downstream tasks, this selected subset achieved an impressive 98.2 percent of the performance of full-data fine-tuning, on average.

Figures (8)

• Figure 1: Previous state-of-the-art methods (e.g., TIVEliu2024less and ICONSwu2024icons) treat importance and diversity as decoupled and independent components, using the entire data in the training stage. Our CoIDO integrates importance and diversity in a coupled and reciprocal optimization, achieving superior data selection by utilizing only a fraction $p\% (p \ll 100 )$ of the full dataset for model training and without any specialized algorithm in the selection stage.
• Figure 2: Framework overview of CoIDO (also see Appendix \ref{['ssec:pseudocode']}). Basic features (image, text, and image-text) are extracted for spectral clustering and scorer training. The plug-in CoIDO Scorer, co-trained with the target MLLM (e.g., LLaVA-1.5-7B), outputs a score $w_i$ for each sample. These scores are used to compute the importance loss $\mathcal{L}_I$ and diversity loss $\mathcal{L}_D$, which are jointly optimized to capture both data importance and diversity via backpropagation using batches of random samples.
• Figure 3: Ablations of different CoIDO Scorer architectures.
• Figure 4: Comparison of different training data ratios $p\%$.
• Figure 5: Performance vs. selection ratios $\gamma$.