MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders

TL;DR

MoVE-KD introduces a framework to distill the strengths of multiple vision encoders into one efficient student by pairing encoder adapters with a Mixture-of-LoRA-Experts (MoLE) and an attention-guided KD loss that leverages CLIP's [CLS] attention to weight both teachers and tokens. The method mitigates conflicts among diverse teachers while keeping parameter growth small, using three core components: encoder adapters, MoLE for selective specialization, and attention-based regularization with a combined loss $\mathcal{L}_{total} = \mathcal{L}_{text} + \lambda_{kd} \cdot \mathcal{L}_{kd}$. Empirical results on LLaVA and LLaVA-NeXT demonstrate state-of-the-art performance on major VLM benchmarks, with ablations underscoring the value of encoder adapters, MoLE, and adaptive teacher/token weighting. The work highlights practical implications for scalable, efficient VLMs and points to future directions in optimizing the projection between visual and language spaces for further gains.

Abstract

Visual encoders are fundamental components in vision-language models (VLMs), each showcasing unique strengths derived from various pre-trained visual foundation models. To leverage the various capabilities of these encoders, recent studies incorporate multiple encoders within a single VLM, leading to a considerable increase in computational cost. In this paper, we present Mixture-of-Visual-Encoder Knowledge Distillation (MoVE-KD), a novel framework that distills the unique proficiencies of multiple vision encoders into a single, efficient encoder model. Specifically, to mitigate conflicts and retain the unique characteristics of each teacher encoder, we employ low-rank adaptation (LoRA) and mixture-of-experts (MoEs) to selectively activate specialized knowledge based on input features, enhancing both adaptability and efficiency. To regularize the KD process and enhance performance, we propose an attention-based distillation strategy that adaptively weighs the different encoders and emphasizes valuable visual tokens, reducing the burden of replicating comprehensive but distinct features from multiple teachers. Comprehensive experiments on popular VLMs, such as LLaVA and LLaVA-NeXT, validate the effectiveness of our method. Our code is available at: https://github.com/hey-cjj/MoVE-KD.

Figures (5)

• Figure 1: Comparison of LLaVA-1.5-7B liu2023improvedllava and RADIO ranzinger2024radio on a wide range of benchmarks, and MoVE-KD surpasses them.
• Figure 2: Concentration of [CLS] attention. The left subfigure (a) is the input image, and the right subfigure (b) is the [CLS] attention visualization of the pre-trained CLIP, showing CLIP's focus on valuable regions of the image.
• Figure 3: The pipeline of MoVE-KD. MoVE-KD projects teacher encoders' outputs using encoder adapters, assigns teacher weight and token weight based on CLIP's [CLS] attention. To mitigate knowledge conflicts, we incorporates MoLE structure in the student encoder.
• Figure 4: The generation of token weight. We employ the attention map of the [CLS] token to measure the contribution of vision tokens to knowledge distillation.
• Figure 5: The visualization of CLIP [CLS] attention and student [CLS] attention.