EmoLat: Text-driven Image Sentiment Transfer via Emotion Latent Space

TL;DR

EmoLat addresses the problem of text-driven image sentiment transfer without relying on reference target images. The authors introduce EmoLat, an emotion latent space built from an emotion semantic graph and GAN-based codebooks, and pair it with an EmoLat-based cross-modal sentiment transfer network that fuses text semantics with visual features via a multi-modal Transformer. They also construct Emospace Set, a large-scale, richly annotated dataset to support learning and evaluation. Experiments on Emospace Set show superior sentiment transfer accuracy and image quality compared to state-of-the-art baselines, enabling controllable, text-guided image editing and pushing forward cross-modal sentiment reasoning.

Abstract

We propose EmoLat, a novel emotion latent space that enables fine-grained, text-driven image sentiment transfer by modeling cross-modal correlations between textual semantics and visual emotion features. Within EmoLat, an emotion semantic graph is constructed to capture the relational structure among emotions, objects, and visual attributes. To enhance the discriminability and transferability of emotion representations, we employ adversarial regularization, aligning the latent emotion distributions across modalities. Building upon EmoLat, a cross-modal sentiment transfer framework is proposed to manipulate image sentiment via joint embedding of text and EmoLat features. The network is optimized using a multi-objective loss incorporating semantic consistency, emotion alignment, and adversarial regularization. To support effective modeling, we construct EmoSpace Set, a large-scale benchmark dataset comprising images with dense annotations on emotions, object semantics, and visual attributes. Extensive experiments on EmoSpace Set demonstrate that our approach significantly outperforms existing state-of-the-art methods in both quantitative metrics and qualitative transfer fidelity, establishing a new paradigm for controllable image sentiment editing guided by textual input. The EmoSpace Set and all the code are available at http://github.com/JingVIPLab/EmoLat.

Figures (5)

• Figure 1: In (a), two images with different objects and corresponding attributes are categorized the same emotion "contentment"; In (b), two images with different emotions have similar objects with different attributes.
• Figure 2: (a) The structure diagram of emotion semantic graph encoder. (b) The structure diagram of the emotion latent space generator is as follows: images are fed into a frozen VGG encoder to extract features, which serve as real data for the network. After extracting features from the emotion semantic map through the encoder, the data is processed by one of codebooks to obtain generated data. The discriminator achieves the feature distribution alignment between the generated data and real data by optimizing the loss function.
• Figure 3: t-SNE Visualization of EmoLat.
• Figure 4: EmoLat based cross-modal sentiment transfer network training framework.
• Figure 5: Qualitative Comparison of Different Sentiment Transfer Methods.