Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Emotional Images: Assessing Emotions in Images and Potential Biases in Generative Models

Maneet Mehta, Cody Buntain

TL;DR

The paper addresses potential negative-emotion bias in AI-generated imagery and its amplification in online media. It evaluates three emotion-recognition strategies (zero-shot vision-language, fine-tuned vision models, and auto-captioning) on EmoSet, finding that fine-tuned ViT yields the strongest performance ($F_1$ up to $0.7343$). Cross-modal analysis shows generated images skew toward negative emotions, notably fear, compared with the associated prompts, a pattern observed across Stable Diffusion and GPT-4o workflows. The work suggests a multidisciplinary path to align AI emotion understanding with psychology and proposes new datasets and evaluation frameworks to better capture multi-label emotional salience. Overall, the findings highlight a potential anti-social feedback loop in generative AI outputs and emphasize the need for safeguards and methodological advances to reduce negative-emotion bias in digital media.

Abstract

This paper examines potential biases and inconsistencies in emotional evocation of images produced by generative artificial intelligence (AI) models and their potential bias toward negative emotions. In particular, we assess this bias by comparing the emotions evoked by an AI-produced image to the emotions evoked by prompts used to create those images. As a first step, the study evaluates three approaches for identifying emotions in images -- traditional supervised learning, zero-shot learning with vision-language models, and cross-modal auto-captioning -- using EmoSet, a large dataset of image-emotion annotations that categorizes images across eight emotional types. Results show fine-tuned models, particularly Google's Vision Transformer (ViT), significantly outperform zero-shot and caption-based methods in recognizing emotions in images. For a cross-modality comparison, we then analyze the differences between emotions in text prompts -- via existing text-based emotion-recognition models -- and the emotions evoked in the resulting images. Findings indicate that AI-generated images frequently lean toward negative emotional content, regardless of the original prompt. This emotional skew in generative models could amplify negative affective content in digital spaces, perpetuating its prevalence and impact. The study advocates for a multidisciplinary approach to better align AI emotion recognition with psychological insights and address potential biases in generative AI outputs across digital media.

Emotional Images: Assessing Emotions in Images and Potential Biases in Generative Models

TL;DR

The paper addresses potential negative-emotion bias in AI-generated imagery and its amplification in online media. It evaluates three emotion-recognition strategies (zero-shot vision-language, fine-tuned vision models, and auto-captioning) on EmoSet, finding that fine-tuned ViT yields the strongest performance ( up to ). Cross-modal analysis shows generated images skew toward negative emotions, notably fear, compared with the associated prompts, a pattern observed across Stable Diffusion and GPT-4o workflows. The work suggests a multidisciplinary path to align AI emotion understanding with psychology and proposes new datasets and evaluation frameworks to better capture multi-label emotional salience. Overall, the findings highlight a potential anti-social feedback loop in generative AI outputs and emphasize the need for safeguards and methodological advances to reduce negative-emotion bias in digital media.

Abstract

This paper examines potential biases and inconsistencies in emotional evocation of images produced by generative artificial intelligence (AI) models and their potential bias toward negative emotions. In particular, we assess this bias by comparing the emotions evoked by an AI-produced image to the emotions evoked by prompts used to create those images. As a first step, the study evaluates three approaches for identifying emotions in images -- traditional supervised learning, zero-shot learning with vision-language models, and cross-modal auto-captioning -- using EmoSet, a large dataset of image-emotion annotations that categorizes images across eight emotional types. Results show fine-tuned models, particularly Google's Vision Transformer (ViT), significantly outperform zero-shot and caption-based methods in recognizing emotions in images. For a cross-modality comparison, we then analyze the differences between emotions in text prompts -- via existing text-based emotion-recognition models -- and the emotions evoked in the resulting images. Findings indicate that AI-generated images frequently lean toward negative emotional content, regardless of the original prompt. This emotional skew in generative models could amplify negative affective content in digital spaces, perpetuating its prevalence and impact. The study advocates for a multidisciplinary approach to better align AI emotion recognition with psychological insights and address potential biases in generative AI outputs across digital media.

Paper Structure

This paper contains 28 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Datasets for Images, Emotion, and AI Prompts
  5. Emotion Identification in Images
  6. Zero-Shot Learning with Vision-Language Models
  7. Fine-Tuning Vision Models for Classification
  8. Auto-Captioning Models
  9. Down-Sampling to Minimize GPT Costs
  10. Validating Automatic Emotion Recognition in Images
  11. Emotion Identification in Text
  12. Comparing Emotion Across Modalities
  13. Results
  14. Emotion Classification in Images
  15. Comparing BLIP2 versus GPT-4o
  16. ...and 13 more sections

Figures (3)

  • Figure 1: Cross-Modality Spearman $\rho$ Correlations Between DeMuX and EmoSet. Results show general alignment for the negative emotions (anger, disgust, fear, and sadness). Alignment is less clear in the positive emotions, where joy in prompts corresponds to both amusement and contentment in images, and excited correlates with awe, but surprise and amusement in prompts is less correlated with the positive emotions in images.
  • Figure 2: Cross-Modality Spearman $\rho$ Correlations Between Amazon's SageMaker and EmoSet. Negative emotions appear similarly aligned compared to DeMuX, but positive emotions show less alignment.
  • Figure 3: EmoSet Images and Single Emotion Labels.