COMPACT: COMPositional Atomic-to-Complex Visual Capability Tuning

TL;DR

COMPACT tackles data efficiency in visual instruction tuning by elevating per-sample complexity through a compositional approach over atomic visual capabilities. It defines a $k$-value complexity metric and a four-step data recipe to generate multi-capability questions, enabling high information density with far fewer examples. Empirical results across eight multimodal benchmarks show COMPACT matches or exceeds full VIT performance using roughly 10% of the data (100.2% relative) and delivers pronounced gains on complex tasks like MM-Vet and MMStar, demonstrating the practicality of complexity-aware data curation. The work highlights significant potential for scalable, data-efficient learning in vision-language systems while outlining limitations and directions for extending the approach to knowledge-intensive tasks and broader capability sets.

Abstract

Visual instruction tuning (VIT) datasets are constructed from randomly sampled image-question pairs, without regard to the informativeness of each pair. Recent dataset selection methods have shown that a small fraction of such datasets enriched with informative samples can lead to efficient finetuning of Multimodal Large Language Models. In this work, we explore the impact of sample complexity on informative data curation and introduce COMPACT (COMPositional Atomic-to-complex Visual Capability Tuning), a VIT data recipe that scales training sample complexity by combining multiple atomic visual capabilities in a single training example. Concretely, we synthesize rich and informative text questions for each image, allowing us to significantly reduce the number of training examples required for effective visual instruction tuning. COMPACT demonstrates superior data efficiency compared to existing data reduction methods. When applied to the LLAVA-665K VIT dataset, COMPACT reduces the data budget by 90% while still achieving 100.2% of the full VIT performance (compared to only 97.5% by the state-of-the-art method) across eight multimodal benchmarks. Further, training on the COMPACT data outperforms training on the full-scale data on particularly complex benchmarks such as MM-Vet (+8.6%) and MMStar (+2.9%). COMPACT offers a scalable and efficient synthetic data generation recipe to improve on visual language tasks.

Figures (13)

• Figure 1: Complexity k. We show that increasing the complexity of LLaVA-665K improves performance. (left) Examples of questions with different k-values, where k is the number of atomic capabilities required. (middle) Distribution of k-value in VIT subset (LLaVA) and VIT subset augmented with 1 additional capability (LLaVAk+1). (right) Performance on downstream tasks (ยง4.1) for VIT subset (LLaVA), VIT subset regenerated with no capability augmentation (LLAVAk+0) and VIT subset augmented with 1 additional capability (LLaVAk+1).
• Figure 2: COMPACT data generation pipeline.(Left): We design a data recipe that can scale the complexity of each training example. We randomly sample $k_{gen} \in \{1,2,3\}$ atomic capabilities such as color, object recognition, and spatial relationship. (Center): We generate questions that integrate all $k_{gen}$ sampled capabilities and verify their quality. (Right): We combine the synthetically generated compositional tuning data with instruction tuning data for response formatting.
• Figure 3: Performance across compositional tuning data scales. We show that COMPACT's compositional tuning data scales more efficiently than conventional VIT. We fix the VIT subset (5% of LLaVA-665Kllava15) and scale the compositional tuning data in COMPACT from 2K to 32K. We compare each mix with VIT only datasets with equal data budgets. COMPACT (solid lines) consistently outperforms LLaVA-665K VIT (dashed lines) with fewer data. COMPACT's compositional tuning data scales particularly well on SeedBench2Plus seedbench, which consists of spatially complex tasks of navigating charts and maps.
• Figure 4: Impact of k-value range: Performance comparison across variations of COMPACT whose compositional tuning data is synthesized with different ranges of $k_{gen}$ in the Capability Sampling step (§\ref{['subsec:data_recipe']}). Models trained on higher and wider ranges of complexity (darker bars) achieve higher performance across benchmarks.
• Figure 5: Impact of instruction tuning data ratio. Relative performance of COMPACT with different amounts instruction tuning data from LLaVA-665Kllava15. The x-axis is the percentage of LLaVA-665K used as instruction tuning data, and the y-axis is relative score. The performance improves significantly with a small amount of instruction tuning data and stabilizes around 5%.