LLAVADI: What Matters For Multimodal Large Language Models Distillation

TL;DR

This paper investigates what matters in distilling multimodal large language models (MLLMs) by introducing LLAVADI, a teacher–student framework that distills from a 13B LLaVA teacher to a 2.7B MobileLLaMA student. It systematically studies four distillation axes—feature embeddings, logits, affinity, and data-driven augmentation—revealing that final-layer hidden embedding alignment and logit-level KL-based distillation are especially impactful, while affinity-based methods provide limited benefit under autoregressive loss. Data-driven augmentation, particularly teacher-generated instruction data, substantially boosts performance, and scaling to 2M instruction-tuning samples yields notable gains. The results show that a carefully designed distillation strategy can let a relatively small model approach or match larger MLLMs on several benchmarks, highlighting practical pathways for efficient deployment and future multimodal knowledge transfer research. The authors also provide code and models to facilitate ongoing exploration in multimodal distillation.

Abstract

The recent surge in Multimodal Large Language Models (MLLMs) has showcased their remarkable potential for achieving generalized intelligence by integrating visual understanding into Large Language Models.Nevertheless, the sheer model size of MLLMs leads to substantial memory and computational demands that hinder their widespread deployment. In this work, we do not propose a new efficient model structure or train small-scale MLLMs from scratch. Instead, we focus on what matters for training small-scale MLLMs through knowledge distillation, which is the first step from the multimodal distillation perspective. Our extensive studies involve training strategies, model choices, and distillation algorithms in the knowledge distillation process. These results show that joint alignment for both tokens and logit alignment plays critical roles in teacher-student frameworks. In addition, we draw a series of intriguing observations from this study. By evaluating different benchmarks and proper strategy, even a 2.7B small-scale model can perform on par with larger models with 7B or 13B parameters. Our code and models will be publicly available for further research.

Figures (2)

• Figure 1: Illustration of various knowledge distillation approaches proposed in this work.
• Figure 2: Comparison of original instruction-tuning data and data generated by the teacher model.