Evaluating Vision-Language Models for Emotion Recognition

TL;DR

This work provides the first comprehensive benchmark (EvE) to evaluate how vision-language systems recognize emotions evoked by images. Using a zero-shot setup across seven VLMs and four diverse datasets, the study assesses correctness and robustness to prompt variations, revealing significant limitations in evoked emotion recognition and a pronounced sensitivity to prompt structure and personas. Through detailed error analysis and a human study, it identifies data quality, label granularity, and subjective interpretation as key sources of misalignment, while offering practical recommendations for dataset design and evaluation practices. The findings highlight the need for robust, interpretable benchmarks and careful data curation to advance empathetic, safe multimodal AI systems in real-world interactions.

Abstract

Large Vision-Language Models (VLMs) have achieved unprecedented success in several objective multimodal reasoning tasks. However, to further enhance their capabilities of empathetic and effective communication with humans, improving how VLMs process and understand emotions is crucial. Despite significant research attention on improving affective understanding, there is a lack of detailed evaluations of VLMs for emotion-related tasks, which can potentially help inform downstream fine-tuning efforts. In this work, we present the first comprehensive evaluation of VLMs for recognizing evoked emotions from images. We create a benchmark for the task of evoked emotion recognition and study the performance of VLMs for this task, from perspectives of correctness and robustness. Through several experiments, we demonstrate important factors that emotion recognition performance depends on, and also characterize the various errors made by VLMs in the process. Finally, we pinpoint potential causes for errors through a human evaluation study. We use our experimental results to inform recommendations for the future of emotion research in the context of VLMs.

Figures (22)

• Figure 1: Positive and Negative Bias demonstrated by the models in simple multimodal classification. Results are averaged across datasets and model sizes.
• Figure 2: (a): The weighted F1 score for each model, averaged across datasets. The different bars represent the orders in which emotion class labels are included in the prompt. (b): The positive and negative sentiment bias, for each model, with different shuffled orders of emotion classes in the prompts.
• Figure 3: The weighted F1 score with and without precise target labels in the prompts. The numbers in brown represent the percentage of fine-grained predictions made.
• Figure 4: Sentiment bias for responses generated without explicit target labels in the prompts.
• Figure 5: Fig. (a): Weighted F1 score for each model, averaged across all datasets considered. The score drops sharply when the models assume any sentimental persona. Fig. (b): Change in Positive Bias when assuming any persona. Positive Bias is increased and decreased significantly by Positive or Negative Persona. Fig. (c): Change in Negative Bias when assuming any persona. Negative Bias is sharply increased when assuming a negative persona but reduced only marginally by positive persona.