EmoCtrl: Controllable Emotional Image Content Generation

TL;DR

EmoCtrl addresses the challenge of jointly controlling content and emotional expression in image generation by introducing two complementary emotion-enhancement modules (Textual and Visual Emotion Enhancement) and learnable emotion tokens for both text and vision. It is built on two tailored datasets, EmoSet+ and EmoEditSet+, which provide content-emotion quadruplets to supervise alignment between semantic content and affective cues. The approach leverages LoRA-fine-tuned LLMs and cross-attention mechanisms to inject emotion knowledge at both the textual and visual levels, optimizing dedicated losses for each modality. Empirical results, including quantitative metrics (Emo-A, EC-A, LPIPS, Sem-C) and a user study, show EmoCtrl achieves superior joint content-emotion fidelity and expressive power, with extensions to stylized and multi-emotion generation that underscore its practical potential and robustness.

Abstract

An image conveys meaning through both its visual content and emotional tone, jointly shaping human perception. We introduce Controllable Emotional Image Content Generation (C-EICG), which aims to generate images that remain faithful to a given content description while expressing a target emotion. Existing text-to-image models ensure content consistency but lack emotional awareness, whereas emotion-driven models generate affective results at the cost of content distortion. To address this gap, we propose EmoCtrl, supported by a dataset annotated with content, emotion, and affective prompts, bridging abstract emotions to visual cues. EmoCtrl incorporates textual and visual emotion enhancement modules that enrich affective expression via descriptive semantics and perceptual cues. The learned emotion tokens exhibit complementary effects, as demonstrated through ablations and visualizations. Quantatitive and qualatitive experiments demonstrate that EmoCtrl achieves faithful content and expressive emotion control, outperforming existing methods across multiple aspects. User studies confirm EmoCtrl's strong alignment with human preference. Moreover, EmoCtrl generalizes well to creative applications, further demonstrating the robustness and adaptability of the learned emotion tokens.

Figures (8)

• Figure 1: Controllable Emotional Image Content Generation with EmoCtrl. Given a content condition ("Ocean") and a target emotion ("Contentment"), EmoCtrl generates images that maintain semantic content while vividly expressing diverse emotional tones.
• Figure 2: SD lacks emotional expressiveness, EmoGen struggles with content control, and LLM + SD produces ambiguous semantics. In contrast, EmoCtrl generates images that faithfully align with both emotion and content conditions.
• Figure 3: Data construction. EmoSet+ and EmoEditSet+ are constructed with quadruplets containing image, emotion label, content label, and affective prompt, serving as data foundation for C-EICG.
• Figure 4: Overview of EmoCtrl. (a) Textual Emotion Enhancement: Emotion tokens are fused with content text in the LLM to enhance emotion at the semantic level. (b) Visual Emotion Enhancement: Emotion tokens are cross-attended within Stable Diffusion to produce affective images through visual cues.
• Figure 5: Comparison with state-of-the-art methods, showing EmoCtrl is superior in both content preservation and emotion expressiveness.