EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuning

TL;DR

EmoVIT tackles the challenge of emotion-aware understanding in vision by introducing a GPT-assisted pipeline that generates emotion-centric instruction data and an emotion-focused Visual Instruction Tuning framework built on InstructBLIP. The approach leverages a dedicated Q-Former to fuse emotion instruction tokens with image embeddings, enabling improved emotion classification, affective reasoning, and humor understanding, while reducing the need for explicit supervision. Across held-in and held-out evaluations on EmoSet and other emotion datasets, EmoVIT demonstrates robust generalization and transferability to other VIT models like LLaVA, with evidence that larger data scales further boost performance. This work establishes a new paradigm for Emotion Visual Instruction Tuning and offers practical routes toward scalable, instruction-driven emotion understanding in vision-language systems.

Abstract

Visual Instruction Tuning represents a novel learning paradigm involving the fine-tuning of pre-trained language models using task-specific instructions. This paradigm shows promising zero-shot results in various natural language processing tasks but is still unexplored in vision emotion understanding. In this work, we focus on enhancing the model's proficiency in understanding and adhering to instructions related to emotional contexts. Initially, we identify key visual clues critical to visual emotion recognition. Subsequently, we introduce a novel GPT-assisted pipeline for generating emotion visual instruction data, effectively addressing the scarcity of annotated instruction data in this domain. Expanding on the groundwork established by InstructBLIP, our proposed EmoVIT architecture incorporates emotion-specific instruction data, leveraging the powerful capabilities of Large Language Models to enhance performance. Through extensive experiments, our model showcases its proficiency in emotion classification, adeptness in affective reasoning, and competence in comprehending humor. The comparative analysis provides a robust benchmark for Emotion Visual Instruction Tuning in the era of LLMs, providing valuable insights and opening avenues for future exploration in this domain. Our code is available at \url{https://github.com/aimmemotion/EmoVIT}.

Figures (11)

• Figure 1: Illustration of the importance of instruction-following ability in visual emotion understanding.
• Figure 2: Performance comparison on EmoSet test set EmoSet (Accuracy %).
• Figure 3: The comparison of different visual tuning paradigms.
• Figure 4: The overall architecture of our proposed method. The Emotion Instruction data generated by (a) will be used for Emotion Visual Instruction Tuning in (b). During Emotion Visual Instruction Tuning, given an input image, the frozen Image Encoder initiates the process by extracting visual features. Emotion Instruction generated by (a) are subsequently interacting with Queries embedding through the learnable Q-Former. This interaction is key to drawing out image features that are relevant to the task at hand. As a result, the frozen LLM receives visual information conducive to instruction following.
• Figure 5: The sample of our generated visual emotion instruction data.