Active Data Curation Effectively Distills Large-Scale Multimodal Models

TL;DR

The paper tackles the challenge of compressing large vision-language models by proposing active data curation as a simple, scalable alternative to traditional knowledge distillation. It introduces ACID, an implicit distillation mechanism that uses a large reference model to curate training data, and ACED, which combines ACID with an explicit distillation loss to achieve superior FLOP efficiency. Across extensive experiments on 27 StableEval tasks and downstream LiT-Decoder benchmarks, ACED sets a new state-of-the-art for inference-efficient zero-shot classification and image-text retrieval, while also delivering stronger vision encoders for generation tasks. The results demonstrate that active data selection, especially with large reference models, can complement and surpass conventional KD in multimodal pretraining, with practical impact for deploying capable, efficient multimodal systems.

Abstract

Knowledge distillation (KD) is the de facto standard for compressing large-scale models into smaller ones. Prior works have explored ever more complex KD strategies involving different objective functions, teacher-ensembles, and weight inheritance. In this work we explore an alternative, yet simple approach -- active data curation as effective distillation for contrastive multimodal pretraining. Our simple online batch selection method, ACID, outperforms strong KD baselines across various model-, data- and compute-configurations. Further, we find such an active data curation strategy to in fact be complementary to standard KD, and can be effectively combined to train highly performant inference-efficient models. Our simple and scalable pretraining framework, ACED, achieves state-of-the-art results across 27 zero-shot classification and retrieval tasks with upto 11% less inference FLOPs. We further demonstrate that our ACED models yield strong vision-encoders for training generative multimodal models in the LiT-Decoder setting, outperforming larger vision encoders for image-captioning and visual question-answering tasks.

Figures (10)

• Figure 1: Performance-Inference Frontier. Our ACED models (Active Curation with Explicit Distillation, see \ref{['sec:methods']}), achieve a new pareto frontier for performance (measured by ImageNet top-1 zero-shot validation accuracy) vs. inference GFLOPs.
• Figure 2: Different Method Configurations. We depict all the different method configurations that we consider in our work. Each method can be independently recovered from the unified objective $\mathcal{L}_\text{full}$ in \ref{['combination-section']}. The iid-sample and acid-sample boxes denote the IID-sampling and our ACID online batch-selection sampling schemes respectively. For more details, refer to \ref{['sec:methods']}.
• Figure 3: StableEval: a reliable set of multimodal evaluations. (left) Variability across random pretraining seeds of individual evaluations. (right) Variability of average performance across incrementally larger sets of evaluations, starting from the most reliable.
• Figure 4: Scaling behaviour of ACID.(left) We scale up the reference model used for training each student (Ti, S and B) with H-ACID---there is an optimal scaling relationship (best reference for each student marked with $\star$) between student and reference sizes. (right) Our H-ACID and I-ACID comprehensively outperform Softmax-KD across all teacher scales. Importantly, our ACIDs outperform the IID baseline even for tiny reference models, whereas Softmax-KD struggles to improve over IID with smaller teachers.
• Figure 5: ACID significantly outperforms KD.(left) We vary the training dataset of the reference/teacher model, and use the same pretrained model as the reference for ACID and teacher for KD---across all configurations, we note strong gains for ACID. (center) Across different distillation objectives and a full hyperparameter sweep for optimal KD conditions, ACID is still the best performing method by large margins. (right)ACID further outperforms KD across three different student sizes.