CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs

TL;DR

The paper addresses the challenge of distilling visual perception in multimodal LLMs by identifying visual attention misalignment as a bottleneck. It introduces CompoDistill, a framework that combines Visual ATtention alignment (VAT) with Teacher Adapter Fetch (TAF) and a three-stage distillation pipeline to transfer both visual recognition and perception from a large teacher to a smaller student. Empirical results show substantial gains on compositional reasoning benchmarks while preserving VQA performance, and the method generalizes to stronger backbones and different backbones, demonstrating data-efficient and robust distillation. This work advances practical, efficient MLLMs by focusing on attention dynamics as a core mechanism for transferring complex visual reasoning capabilities.

Abstract

Recently, efficient Multimodal Large Language Models (MLLMs) have gained significant attention as a solution to their high computational complexity, making them more practical for real-world applications. In this regard, the knowledge distillation (KD) approach has emerged as a promising alternative, which transfers the rich visual and linguistic knowledge from a larger model (teacher) to a smaller model (student). However, we observe that existing KD methods struggle to effectively distill the teacher MLLM's rich visual perception abilities to the student, a challenge that has been largely overlooked in previous studies. Through a systematic analysis, we identify visual attention misalignment between student and teacher as the main cause of this issue. Based on this insight, we propose CompoDistill, a novel KD framework that explicitly aligns the student's visual attention with that of the teacher to enhance the student's visual perception abilities. Our extensive experiments show that CompoDistill significantly improves performance on compositional reasoning tasks that require visual perception abilities while maintaining strong performance on visual question answering tasks, as done in existing studies. Furthermore, CompoDistill demonstrates effectiveness with a more advanced backbone, highlighting its generalizability.

Figures (9)

• Figure 1: (a) Comparison of the teacher (LLaVA-4B), SFT (LLaVA-2B), and various 2B KD methods distilled from the same teacher. (b) Visual attention misalignment between the student (LLaVA-KD-2B) and teacher (LLaVA-4B), where the student attends to irrelevant image regions for the text query A woman is on the table, in contrast to the teacher. For more examples including our proposed method, CompoDistill, please refer to Appendix \ref{['app_sec:attn_misalign']}.
• Figure 2: (a) Attention of the answer token over visual tokens and text tokens during its generation. The transparency indicates the degree of attention the answer token gives to these tokens. (b-e) Layer-wise Teacher-Student and Teacher-SFT attention similarities over visual tokens ((b), (c)) and text tokens ((d), (e)). GQA is used for VQA ((b), (d)), and SugarCrepe is used for CR ((c), (e)). Results on other datasets are shown in Appendix \ref{['app_sec:additional_similar']}.
• Figure 3: Relationship between attention similarity and performance on the GQA (VQA) dataset.
• Figure 4: Change in performance the student attention is substituted with the teacher attention on the SugarCrepe (CR) dataset.
• Figure 5: Overview of CompoDistill. It consists of the VAT module and the TAF module.