EmoGen: Emotional Image Content Generation with Text-to-Image Diffusion Models

TL;DR

This work introduces Emotional Image Content Generation (EICG), a new task to generate semantic-clear and emotion-faithful images given emotion categories, and proposes an emotion space and construct a mapping network to align it with the powerful Contrastive Language-Image Pre-training (CLIP) space, providing a concrete interpretation of abstract emotions.

Abstract

Recent years have witnessed remarkable progress in image generation task, where users can create visually astonishing images with high-quality. However, existing text-to-image diffusion models are proficient in generating concrete concepts (dogs) but encounter challenges with more abstract ones (emotions). Several efforts have been made to modify image emotions with color and style adjustments, facing limitations in effectively conveying emotions with fixed image contents. In this work, we introduce Emotional Image Content Generation (EICG), a new task to generate semantic-clear and emotion-faithful images given emotion categories. Specifically, we propose an emotion space and construct a mapping network to align it with the powerful Contrastive Language-Image Pre-training (CLIP) space, providing a concrete interpretation of abstract emotions. Attribute loss and emotion confidence are further proposed to ensure the semantic diversity and emotion fidelity of the generated images. Our method outperforms the state-of-the-art text-to-image approaches both quantitatively and qualitatively, where we derive three custom metrics, i.e., emotion accuracy, semantic clarity and semantic diversity. In addition to generation, our method can help emotion understanding and inspire emotional art design.

Figures (8)

• Figure 1: Emotional Image Content Generation (EICG). Given an emotion category, our network produces images that exhibit unambiguous meanings (semantic-clear), reflect the intended emotion (emotion-faithful) and incorporate varied semantics (semantic-diverse).
• Figure 2: Despite (b) CLIP space demonstrates a powerful semantic structure, it struggles to effectively capture emotional relationships within (a), the proposed emotion space.
• Figure 3: Training process of our network. Emotion representation (stage 1) learns a well-behaved emotion space and emotion content generation (stage 2) maps this space to CLIP space, aiming to generate image contents with emotion fidelity, semantic clarity and diversity.
• Figure 4: Motivation for loss function design. Compare to (a) LDM loss alone, (b) attribute loss enhances semantic clarity while (c) emotion confidence ensures emotion accuracy.
• Figure 5: Illustration of emotion confidence. Each (a) attribute is represented by (b) a distribution of confidence on eight emotions.