Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection

TL;DR

Adapt-infinity addresses the challenge of lifelong multimodal instruction tuning (LiIT) by reframing data selection as a dynamic problem over an expanding dataset pool. It introduces gradient-based pseudo-task clustering and a multi-way, entropy-guided data selector, augmented by a novel Image Grounding score to emphasize truly multimodal samples. A combined permanent-temporal pruning strategy controls dataset growth, enabling scalable training while mitigating forgetting and promoting forward transfer. Across diverse multimodal tasks on LLaVA-1.5, Adapt-infinity achieves strong retention with a fraction of the data, outperforming standard replay and score-based pruning baselines and demonstrating robustness to dataset scale.

Abstract

Visual instruction datasets from various distributors are released at different times and often contain a significant number of semantically redundant text-image pairs, depending on their task compositions (i.e., skills) or reference sources. This redundancy greatly limits the efficient deployment of continually adaptable multimodal large language models, hindering their ability to refine existing skills and acquire new competencies over time. We reframe the problem of lifelong Instruction Tuning (LiIT) via data selection, where the model automatically selects beneficial samples to learn from earlier and new datasets based on the current state of acquired knowledge in the model. We propose Adapt-$\infty$, a new multi-way and adaptive data selection approach that dynamically balances sample efficiency and effectiveness during LiIT. We first construct pseudo-skill clusters by grouping gradient-based sample vectors. Next, we select the best-performing data selector for each skill cluster from a pool of selector experts, including our newly proposed scoring function, Image Grounding score. This data selector samples a subset of the most important samples from each skill cluster for training. To prevent the continuous increase in the size of the dataset pool during LiIT, we introduce a cluster-wise permanent data pruning strategy to remove the most semantically redundant samples from each cluster, keeping computational requirements manageable. We validate the effectiveness and efficiency of Adapt-$\infty$ over a sequence of multimodal instruction tuning datasets with various tasks, including (Knowledge) VQA, multilingual, grounding, reasoning, language-only, and multi-image comprehension. Training with samples selected by Adapt-$\infty$ alleviates catastrophic forgetting, especially for rare tasks, and promotes forward transfer across the continuum using only a fraction of the original data.

Figures (9)

• Figure 1: Illustration of Adapt-$\infty$. When a new dataset is incorporated into the data pool at the beginning of each timestep, Adapt-$\infty$ extracts sample vectors and forms pseudo-task clusters based on their similarity. Using a set of scoring functions, Adapt-$\infty$ predicts the most suitable scoring function for each cluster and trains an MLLM on the selected samples. To prevent excessive computation as the pool size grows, we introduce dataset compression by permanently removing redundant samples.
• Figure 2: A: Sample distributions for different visual language tasks in M3IT li2023m3it based on two importance scores, EL2N and entropy. B: t-SNE visualization of sample vectors based on their gradients and features. C: Histogram of Perplexity and Image Grounding scores. We visualize a few samples from M3IT with prompts (black) and ground-truth answers (blue)
• Figure 3: Average accuracies per skill over time. Comparison of average accuracies over time for each skill in our evaluation suite using various data selection methods. Higher is better.
• Figure 4: Nearest neighbor samples for query samples from VQA (top) and Chinese VQA (bottom) tasks in the gradient (left) and feature spaces (right).
• Figure 5: A: Relative gain % for different skills using various data selection methods in the lifelong multimodal instruction tuning setting. B: t-SNE visualization of gradient vectors of the M3IT dataset from layers of varying depth in the LLaVA model.