Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Visual Affect Analysis: Predicting Emotions of Image Viewers with Vision-Language Models

Filip Nowicki, Hubert Marciniak, Jakub Łączkowski, Krzysztof Jassem, Tomasz Górecki, Vimala Balakrishnan, Desmond C. Ong, Maciej Behnke

TL;DR

This work benchmarks vision–language architectures on three psychometric affective image datasets (IAPS, NAPS, LAI-GAI) to evaluate how well zero-shot VLM outputs align with human ratings across discrete emotions and continuous affective dimensions. It demonstrates that while VLMs can classify core emotions with reasonable accuracy and correlate with human judgments on intensity, they exhibit systematic biases (notably in arousal) and struggle with anger and surprise, indicating limited nuanced understanding. Rater-conditioned prompting yields only modest, inconsistent gains, suggesting current prompts and demographic conditioning are not robust strategies for improving affective prediction. The study highlights the potential of VLMs as supportive tools for affective science (e.g., coarse annotations and pre-screening) while emphasizing the need for richer datasets, task-specific prompting, and calibration to move toward reliable, scalable affective assessment.

Abstract

Vision-language models (VLMs) show promise as tools for inferring affect from visual stimuli at scale; it is not yet clear how closely their outputs align with human affective ratings. We benchmarked nine VLMs, ranging from state-of-the-art proprietary models to open-source models, on three psycho-metrically validated affective image datasets: the International Affective Picture System, the Nencki Affective Picture System, and the Library of AI-Generated Affective Images. The models performed two tasks in the zero-shot setting: (i) top-emotion classification (selecting the strongest discrete emotion elicited by an image) and (ii) continuous prediction of human ratings on 1-7/9 Likert scales for discrete emotion categories and affective dimensions. We also evaluated the impact of rater-conditioned prompting on the LAI-GAI dataset using de-identified participant metadata. The results show good performance in discrete emotion classification, with accuracies typically ranging from 60% to 80% on six-emotion labels and from 60% to 75% on a more challenging 12-category task. The predictions of anger and surprise had the lowest accuracy in all datasets. For continuous rating prediction, models showed moderate to strong alignment with humans (r > 0.75) but also exhibited consistent biases, notably weaker performance on arousal, and a tendency to overestimate response strength. Rater-conditioned prompting resulted in only small, inconsistent changes in predictions. Overall, VLMs capture broad affective trends but lack the nuance found in validated psychological ratings, highlighting their potential and current limitations for affective computing and mental health-related applications.

Visual Affect Analysis: Predicting Emotions of Image Viewers with Vision-Language Models

TL;DR

This work benchmarks vision–language architectures on three psychometric affective image datasets (IAPS, NAPS, LAI-GAI) to evaluate how well zero-shot VLM outputs align with human ratings across discrete emotions and continuous affective dimensions. It demonstrates that while VLMs can classify core emotions with reasonable accuracy and correlate with human judgments on intensity, they exhibit systematic biases (notably in arousal) and struggle with anger and surprise, indicating limited nuanced understanding. Rater-conditioned prompting yields only modest, inconsistent gains, suggesting current prompts and demographic conditioning are not robust strategies for improving affective prediction. The study highlights the potential of VLMs as supportive tools for affective science (e.g., coarse annotations and pre-screening) while emphasizing the need for richer datasets, task-specific prompting, and calibration to move toward reliable, scalable affective assessment.

Abstract

Vision-language models (VLMs) show promise as tools for inferring affect from visual stimuli at scale; it is not yet clear how closely their outputs align with human affective ratings. We benchmarked nine VLMs, ranging from state-of-the-art proprietary models to open-source models, on three psycho-metrically validated affective image datasets: the International Affective Picture System, the Nencki Affective Picture System, and the Library of AI-Generated Affective Images. The models performed two tasks in the zero-shot setting: (i) top-emotion classification (selecting the strongest discrete emotion elicited by an image) and (ii) continuous prediction of human ratings on 1-7/9 Likert scales for discrete emotion categories and affective dimensions. We also evaluated the impact of rater-conditioned prompting on the LAI-GAI dataset using de-identified participant metadata. The results show good performance in discrete emotion classification, with accuracies typically ranging from 60% to 80% on six-emotion labels and from 60% to 75% on a more challenging 12-category task. The predictions of anger and surprise had the lowest accuracy in all datasets. For continuous rating prediction, models showed moderate to strong alignment with humans (r > 0.75) but also exhibited consistent biases, notably weaker performance on arousal, and a tendency to overestimate response strength. Rater-conditioned prompting resulted in only small, inconsistent changes in predictions. Overall, VLMs capture broad affective trends but lack the nuance found in validated psychological ratings, highlighting their potential and current limitations for affective computing and mental health-related applications.
Paper Structure (30 sections, 1 equation, 4 figures, 4 tables)

This paper contains 30 sections, 1 equation, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Key Evaluation Factor: Affective Image Datasets
  5. Methods
  6. Datasets
  7. International Affective Picture System (IAPS)
  8. Nencki Affective Picture System (NAPS)
  9. Library of AI-Generated Affective Images (LAI-GAI)
  10. Tasks
  11. Task 1: Emotion Classification
  12. Task 2: Affective Dimension Prediction
  13. Task 3: Rater-conditioned Prompting
  14. Models
  15. Proprietary:
  16. ...and 15 more sections

Figures (4)

  • Figure 1: Visualization of the affective stimulus space via t-SNE embeddings. The figure compares ground-truth human emotion ratings with predictions from the Qwen-VL-2.5-72B model across three datasets. The panels are arranged as follows: (a) LAI-GAI, (c) IAPS, and (e) NAPS show the space colored by human-assigned emotion labels. Panels (b) LAI-GAI, (d) IAPS, and (f) NAPS show the same space colored by the model's predicted emotion labels. Each point represents an image, with its position determined by t-SNE applied to its L2-normalized CLIP vision embedding.
  • Figure 2: Three pairs of visually similar images eliciting different emotions. The pairs were identified in the IAPS dataset; however, due to copyright restrictions, we generated similar images using AI models (GPT Image openai_gpt_image_1 for a, b, e; Nano Banana google_gemini2.5flash_2025 for c, d, f). Each original pair is intended to elicit discrete and contrasting emotions, as shown in the subcaptions (a-f).
  • Figure 3: Effect of participant background information on prediction accuracy for Gemini-2.5-Flash (a, c, e) and Gemma-3 27B (b, d, f). Rows denote background type. Plots report Cohen’s $d$ for the changes in MAE (positive = improved accuracy); error bars show 95% CIs, with effects significant when the CI excludes zero.
  • Figure 4: Comparison of mean emotion ratings across datasets. Each panel displays a dot plot comparing the mean predicted emotion ratings from the models with the human baseline. The $y$-axis lists the emotions, and the $x$-axis represents the mean rating. Each marker shape and color corresponds to a specific model, as detailed in the legend within the plots. Panels show results for (a) LAI-GAI, (b) NAPS, and (c) IAPS.