Fuzzy-aware Loss for Source-free Domain Adaptation in Visual Emotion Recognition

TL;DR

The paper targets source-free domain adaptation for visual emotion recognition by addressing two forms of label uncertainty: fuzzy emotion labels and fuzzy pseudo-labels. It introduces the fuzzy-aware loss (FAL), which couples the standard cross-entropy with a fuzzy-robust term and, optionally, class-weighting from a memory bank, to better handle uncertain targets without relying on source data. The authors provide a boundedness argument for the fuzzy term and prove a noise-tolerance bound under asymmetric label noise, connecting FAL to focal loss and reverse cross entropy. Empirically, FAL achieves state-of-the-art results on VER benchmarks and remains competitive on standard SFDA tasks like Office-Home, demonstrating both practical effectiveness and generalization. Overall, FAL offers a principled, parameter-free approach to robust domain adaptation in privacy-preserving visual emotion recognition.

Abstract

Source-free domain adaptation in visual emotion recognition (SFDA-VER) is a highly challenging task that requires adapting VER models to the target domain without relying on source data, which is of great significance for data privacy protection. However, due to the unignorable disparities between visual emotion data and traditional image classification data, existing SFDA methods perform poorly on this task. In this paper, we investigate the SFDA-VER task from a fuzzy perspective and identify two key issues: fuzzy emotion labels and fuzzy pseudo-labels. These issues arise from the inherent uncertainty of emotion annotations and the potential mispredictions in pseudo-labels. To address these issues, we propose a novel fuzzy-aware loss (FAL) to enable the VER model to better learn and adapt to new domains under fuzzy labels. Specifically, FAL modifies the standard cross entropy loss and focuses on adjusting the losses of non-predicted categories, which prevents a large number of uncertain or incorrect predictions from overwhelming the VER model during adaptation. In addition, we provide a theoretical analysis of FAL and prove its robustness in handling the noise in generated pseudo-labels. Extensive experiments on 26 domain adaptation sub-tasks across three benchmark datasets demonstrate the effectiveness of our method.

Figures (2)

• Figure 1: The confusion matrix (%) for directly testing the source model (ResNet-50) on the target data domain (EmoSet$\rightarrow$FI).
• Figure 2: Visualization of feature representations learned by the source model (top) and our FAL (bottom) on two tasks of the 8-class emotion datasets.