EM-KD: Distilling Efficient Multimodal Large Language Model with Unbalanced Vision Tokens

TL;DR

Efficient Multimodal Large Language Models (MLLMs) often suffer information loss when vision tokens are aggressively compressed. EM-KD distills knowledge from a vanilla teacher into an Efficient MLLM under unbalanced vision tokens by (i) aligning teacher and student vision tokens with a Hungarian assignment applied to decoded vision logits, (ii) transferring semantic knowledge through Vision Semantic Distillation via reverse KL divergence over the vocabulary, and (iii) strengthening cross-modal alignment with Vision-Language Affinity Distillation. The framework combines these distillation signals with supervised learning on responses to form a weighted objective and is validated across 11 benchmarks, showing substantial accuracy gains and faster inference versus prior efficient MLLMs and distillation methods. The results demonstrate EM-KD’s ability to preserve fine-grained visual understanding while maintaining or improving efficiency, highlighting its practical potential for deploying capable yet resource-efficient MLLMs.

Abstract

Efficient Multimodal Large Language Models (MLLMs) compress vision tokens to reduce resource consumption, but the loss of visual information can degrade comprehension capabilities. Although some priors introduce Knowledge Distillation to enhance student models, they overlook the fundamental differences in fine-grained vision comprehension caused by unbalanced vision tokens between the efficient student and vanilla teacher. In this paper, we propose EM-KD, a novel paradigm that enhances the Efficient MLLMs with Knowledge Distillation. To overcome the challenge of unbalanced vision tokens, we first calculate the Manhattan distance between the vision logits of teacher and student, and then align them in the spatial dimension with the Hungarian matching algorithm. After alignment, EM-KD introduces two distillation strategies: 1) Vision-Language Affinity Distillation (VLAD) and 2) Vision Semantic Distillation (VSD). Specifically, VLAD calculates the affinity matrix between text tokens and aligned vision tokens, and minimizes the smooth L1 distance of the student and the teacher affinity matrices. Considering the semantic richness of vision logits in the final layer, VSD employs the reverse KL divergence to measure the discrete probability distributions of the aligned vision logits over the vocabulary space. Comprehensive evaluation on diverse benchmarks demonstrates that EM-KD trained model outperforms prior Efficient MLLMs on both accuracy and efficiency with a large margin, validating its effectiveness. Compared with previous distillation methods, which are equipped with our proposed vision token matching strategy for fair comparison, EM-KD also achieves better performance.

Figures (4)

• Figure 1: Comparison with existing MLLM distillation approaches. (a) MiniLLM (represent the LLM-style methods) focuses on response token distillation but neglects both the distinctive characteristics of visual features and vision-language correlations. (b) LLaVA-KD (represent the MLLM-style methods) aligns the teacher and student vision representations by modeling the correlations between vision tokens, but is limited to the condition of spatial alignment. (c) Our EM-KD can generalize to scenarios with unbalanced vision tokens between teachers and students, and introducing vision-language affinity distillation further enhances cross-modal alignment.
• Figure 2: We decode each vision token into vocabulary space via LM head, and find vision logits exhibit rich semantic.
• Figure 3: T-SNE visualization of tokens. Vision and textual representations gradually intermix during LLM propagation. Red: system tokens, blue: text tokens, green: vision tokens.
• Figure 4: Framework of the proposed EM-KD. EM-KD consists of three key components: Vision Token Matching, Vision Semantic Distillation and Vision-Language Affinity Distillation. Vision Token Matching employs the Hungarian algorithm to resolve the unbalanced vision tokens problem between the teacher and student. Vision Semantic Distillation performs one-to-one knowledge transfer between matched tokens in logits space. Vision-Language Affinity Distillation further strengthens cross-modal alignment.