Robust Emotion Recognition in Context Debiasing

TL;DR

This work addresses context bias in context-aware emotion recognition (CAER) by introducing CLEF, a causal-debiasing framework that leverages counterfactual inference. CLEF builds a generalized CAER causal graph, adds a non-invasive context branch to capture the direct harmful context effect, and computes debiased predictions by subtracting this direct effect from the total causal effect using counterfactual scenarios. The approach is model-agnostic and demonstrates consistent improvements across multiple CAER backbones on EMOTIC and CAER-S, with ablations validating the necessity of the context branch, KL regularization, and masking. By decoupling good context priors from harmful bias, CLEF offers a principled, scalable path to robust emotion recognition in unconstrained environments, with practical implications for real-world affective computing systems.

Abstract

Context-aware emotion recognition (CAER) has recently boosted the practical applications of affective computing techniques in unconstrained environments. Mainstream CAER methods invariably extract ensemble representations from diverse contexts and subject-centred characteristics to perceive the target person's emotional state. Despite advancements, the biggest challenge remains due to context bias interference. The harmful bias forces the models to rely on spurious correlations between background contexts and emotion labels in likelihood estimation, causing severe performance bottlenecks and confounding valuable context priors. In this paper, we propose a counterfactual emotion inference (CLEF) framework to address the above issue. Specifically, we first formulate a generalized causal graph to decouple the causal relationships among the variables in CAER. Following the causal graph, CLEF introduces a non-invasive context branch to capture the adverse direct effect caused by the context bias. During the inference, we eliminate the direct context effect from the total causal effect by comparing factual and counterfactual outcomes, resulting in bias mitigation and robust prediction. As a model-agnostic framework, CLEF can be readily integrated into existing methods, bringing consistent performance gains.

Figures (6)

• Figure 1: Illustration of the context bias in the CAER task. GT stands for the Ground Truth. Context-specific semantics easily yield spurious shortcuts with emotion labels during training to confound the model mittal2020emoticon, giving erroneous results. Conversely, our CLEF effectively corrects biased predictions.
• Figure 2: We conduct toy experiments to show the effects of context semantics. The indirect effect of the good context prior follows ensemble branches, narrowing the emotion candidate space. The bad direct effect follows the context branch, causing pure bias.
• Figure 3: (a) Examples of a causal graph where nodes represent variables and arrows represent causal effects. (b) Examples of counterfactual notations. (c) The proposed CAER causal graph.
• Figure 4: High-level overview of the proposed CLEF framework implementation. In addition to the vanilla CAER model, we introduce an additional context branch in a non-intrusive manner to capture the pure context bias as the direct context effect. By comparing factual and counterfactual outcomes, our framework effectively mitigates the interference of the harmful bias and achieves debiased emotion inference.
• Figure 5: Emotion classification accuracy (%) for each category of different CLEF-based methods on the CAER-S dataset.