Emotion-Director: Bridging Affective Shortcut in Emotion-Oriented Image Generation

TL;DR

Emotion-Director addresses the gap where emotion is not equivalent to semantics by introducing cross-modal collaboration between a multimodal diffusion model (MC-Diffusion) and a multi-agent prompt-rewriting system (MC-Agent). MC-Diffusion augments textual prompts with emotion-aligned visual prompts and uses an improved DPO loss with negative visual prompts to sharpen emotion discrimination under the same semantics, while MC-Agent generates diverse, visually expressive prompts via a chain-of-concept workflow across multiple agents. Together, they produce emotion-highlighted images with higher emotion faithfulness and visual expressiveness than state-of-the-art baselines, supported by a newly built emotion-oriented training dataset (ETI) and extensive ablations. The work demonstrates strong potential for practical applications in advertising and media, and opens avenues for emotion-aware video generation.

Abstract

Image generation based on diffusion models has demonstrated impressive capability, motivating exploration into diverse and specialized applications. Owing to the importance of emotion in advertising, emotion-oriented image generation has attracted increasing attention. However, current emotion-oriented methods suffer from an affective shortcut, where emotions are approximated to semantics. As evidenced by two decades of research, emotion is not equivalent to semantics. To this end, we propose Emotion-Director, a cross-modal collaboration framework consisting of two modules. First, we propose a cross-Modal Collaborative diffusion model, abbreviated as MC-Diffusion. MC-Diffusion integrates visual prompts with textual prompts for guidance, enabling the generation of emotion-oriented images beyond semantics. Further, we improve the DPO optimization by a negative visual prompt, enhancing the model's sensitivity to different emotions under the same semantics. Second, we propose MC-Agent, a cross-Modal Collaborative Agent system that rewrites textual prompts to express the intended emotions. To avoid template-like rewrites, MC-Agent employs multi-agents to simulate human subjectivity toward emotions, and adopts a chain-of-concept workflow that improves the visual expressiveness of the rewritten prompts. Extensive qualitative and quantitative experiments demonstrate the superiority of Emotion-Director in emotion-oriented image generation.

Figures (8)

• Figure 1: Two cases of emotion-oriented image generation. Given a textual prompt and an intended emotion, our proposed Emotion-Director rewrites the prompt based on the specified emotion, then generates emotion-highlighted images.
• Figure 2: Illustration of affective shortcut, which consists of semantic shortcut and expressive shortcut. (a) Semantic shortcut: Current methods tightly couple emotions with semantic attributes. As a result, they fail to generate emotion-faithful images when the prompt’s semantics fall outside the emotion-specific attributes. Using above prompt as an example, we generate 10 images for each emotion, only 38.75% simultaneously contain a dog and convey the intended emotion. (b) Expressive shortcut: Directly using LLMs to rewrite prompts often leads to template-like outputs that cannot be visually expressed. Using the above prompt as an example, we employ Qwen2.5 to generate 10 rewritten prompts for each emotion. For both the original and rewritten prompts, we use SDXL to generate 4 images and compute the average text–image consistency. The CLIP Score decreases from 35 to 31.
• Figure 3: Pipeline of Emotion-Director. VCE, EA, TPR, and Check represent the four procedures of the MC-Agent: visual concept extraction, emotion attribution, textual prompt revision, and checking. SA and CA refer to self-attention and cross-attention in the diffusion model, respectively. CA-L indicates that LoRA is applied to the cross-attention.
• Figure 4: Qualitative comparison with emotion-oriented baselines. EmoGen aims to generate emotion-oriented images. AIF, EmoEdit, EmoEditor edit a source image into an emotion-oriented image, the source image is generated by SD v1.5. To ensure fairness and controllability, we follow EmoGen's emotion-transfer setting to generate images that match both the textual prompt’s semantics and the intended emotion. All methods use SD v1.5 for fairness, except AIF, which is non-diffusion.
• Figure 5: Qualitative comparison with EmotiCrafter. Both methods use SDXL for fairness.