EmoVLM-KD: Fusing Distilled Expertise with Vision-Language Models for Visual Emotion Analysis

TL;DR

This work tackles visual emotion analysis (VEA) by leveraging complementary strengths of instruction-tuned vision-language models (VLMs) and conventional vision models. It introduces EmoVLM-KD, a three-stage approach that first emotion-instruction-tunes a VLM, then distills domain-specific vision knowledge into a lightweight module attached to the VLM's visual encoder while freezing the VLM, and finally fuses the two signal sources with a gate module. Through KL-divergence-based knowledge distillation and a learnable gate, the method achieves state-of-the-art results across five VE(A) datasets with a minimal parameter increase (~0.003% of the VLM parameters). The approach highlights a practical path to combine general multimodal understanding with specialized visual cues, enhancing robustness and efficiency for emotion recognition in images.

Abstract

Visual emotion analysis, which has gained considerable attention in the field of affective computing, aims to predict the dominant emotions conveyed by an image. Despite advancements in visual emotion analysis with the emergence of vision-language models, we observed that instruction-tuned vision-language models and conventional vision models exhibit complementary strengths in visual emotion analysis, as vision-language models excel in certain cases, whereas vision models perform better in others. This finding highlights the need to integrate these capabilities to enhance the performance of visual emotion analysis. To bridge this gap, we propose EmoVLM-KD, an instruction-tuned vision-language model augmented with a lightweight module distilled from conventional vision models. Instead of deploying both models simultaneously, which incurs high computational costs, we transfer the predictive patterns of a conventional vision model into the vision-language model using a knowledge distillation framework. Our approach first fine-tunes a vision-language model on emotion-specific instruction data and then attaches a distilled module to its visual encoder while keeping the vision-language model frozen. Predictions from the vision language model and the distillation module are effectively balanced by a gate module, which subsequently generates the final outcome. Extensive experiments show that EmoVLM-KD achieves state-of-the-art performance on multiple visual emotion analysis benchmark datasets, outperforming the existing methods while maintaining computational efficiency. The code is available in https://github.com/sange1104/EmoVLM-KD.

Figures (4)

• Figure 1: Performance comparison between Vision Transformer (ViT) and VLM. ViT- and VLM-only represent the proportion of instances correctly predicted by each model individually, whereas the other model fails. Although the overall performance of the two models is similar, the significant proportions of ViT-only and VLM-only suggest the need for the two models to complement each other.
• Figure 2: The overall framework of EmoVLM-KD is trained in three stages. (i) Stage 1: a VLM is trained using three types of emotion instruction data. (ii) Stage 2: ViT serves as the teacher, while the distillation module added to the visual encoder acts as the student, distilling the teacher’s knowledge. (iii) Stage 3: The predictions from the LLM and the distillation module are integrated through a gating mechanism to derive the final output.
• Figure 3: Accuracy (%) and KL divergence loss for different alpha hyperparameter settings.
• Figure 4: Qualitative examples of EmoVLM-KD's prediction.