Bridging Visual Affective Gap: Borrowing Textual Knowledge by Learning from Noisy Image-Text Pairs

TL;DR

This work addresses the gap between pre-trained visual representations and emotion understanding by bridging the visual affective gap with knowledge borrowed from pre-trained textual models. It introduces Partitioned Adaptive Contrastive Learning (PACL), a three-stage framework that partitions noisy image–text data by factual and emotional connections, clusters unimodal samples to obtain pseudo-labels, and applies an adaptive, partition-aware contrastive loss to align visual and textual features. Across six VER benchmarks and multiple backbones, PACL yields consistent improvements over strong baselines and achieves state-of-the-art performance in both standard and zero-shot settings, demonstrating the practical value of text-informed visual emotion perception. The approach requires relatively modest data and computational resources, making it a scalable strategy for enhancing emotional perception in vision systems and advancing robust multimodal emotion understanding.

Abstract

Visual emotion recognition (VER) is a longstanding field that has garnered increasing attention with the advancement of deep neural networks. Although recent studies have achieved notable improvements by leveraging the knowledge embedded within pre-trained visual models, the lack of direct association between factual-level features and emotional categories, called the "affective gap", limits the applicability of pre-training knowledge for VER tasks. On the contrary, the explicit emotional expression and high information density in textual modality eliminate the "affective gap". Therefore, we propose borrowing the knowledge from the pre-trained textual model to enhance the emotional perception of pre-trained visual models. We focus on the factual and emotional connections between images and texts in noisy social media data, and propose Partitioned Adaptive Contrastive Learning (PACL) to leverage these connections. Specifically, we manage to separate different types of samples and devise distinct contrastive learning strategies for each type. By dynamically constructing negative and positive pairs, we fully exploit the potential of noisy samples. Through comprehensive experiments, we demonstrate that bridging the "affective gap" significantly improves the performance of various pre-trained visual models in downstream emotion-related tasks. Our code is released on https://github.com/wdqqdw/PACL.

Figures (4)

• Figure 1: Illustration of the "affective gap" in the pre-trained visual model and how our proposed method bridges it using knowledge from the pre-trained textual model.
• Figure 2: Overview of PACL's three-stage pipeline. In the first stage, we divide the dataset into four partitions according to whether samples are factual-matched, emotional-matched, or not. In the second stage, we perform k-means clustering on image and text separately to obtain the factual and emotional pseudo-label for each of them. In the third stage, we guide the knowledge transfer from text to image with an adaptive contrastive learning. It dynamically constructs positive and negative pairs for different partitions by leveraging the pseudo-labels of samples.
• Figure 3: Samples from four partitions of TumEmo tmm2021tumemo.
• Figure 4: Network architecture of PACL.