Multi-label Class Incremental Emotion Decoding with Augmented Emotional Semantics Learning

TL;DR

This work tackles multi-label class incremental emotion decoding (MLCIL) in dynamic human-computer interaction by introducing the AESL framework, which integrates an augmented emotional relation graph with label disambiguation, semantic-guided feature decoupling, and relation-based knowledge distillation from affective dimensions. A graph-autoencoder-based emotional semantics learning module yields emotion embeddings that guide label-aware feature extraction, while RKD aligns model representations with the affective space to mitigate future-missing labels. The method demonstrates strong, consistent improvements over state-of-the-art baselines across Brain27, Video27, and Audio28 under several incremental protocols, supported by ablations, visualization, and statistical tests. These results suggest AESL’s potential for robust, scalable emotion decoding in real-world, continually evolving affective systems.

Abstract

Emotion decoding plays an important role in affective human-computer interaction. However, previous studies ignored the dynamic real-world scenario, where human experience a blend of multiple emotions which are incrementally integrated into the model, leading to the multi-label class incremental learning (MLCIL) problem. Existing methods have difficulty in solving MLCIL issue due to notorious catastrophic forgetting caused by partial label problem and inadequate label semantics mining. In this paper, we propose an augmented emotional semantics learning framework for multi-label class incremental emotion decoding. Specifically, we design an augmented emotional relation graph module with label disambiguation to handle the past-missing partial label problem. Then, we leverage domain knowledge from affective dimension space to alleviate future-missing partial label problem by knowledge distillation. Besides, an emotional semantics learning module is constructed with a graph autoencoder to obtain emotion embeddings in order to guide the semantic-specific feature decoupling for better multi-label learning. Extensive experiments on three datasets show the superiority of our method for improving emotion decoding performance and mitigating forgetting on MLCIL problem.

Figures (10)

• Figure 1: The illustration of the multi-label class-incremental emotion decoding task. Supposed there are three emotion categories in total: adoration, awe, and joy, which are incrementally learned in three tasks. $\Phi^1$,$\Phi^2$,$\Phi^3$ are the parameters of the unified model $\psi$, which are continuously trained.
• Figure 2: The framework of AESL for multi-label class incremental emotion decoding. (a) Emotional semantics learning and semantic-guided feature decoupling procedure in incremental learning scenario. We omit semantic-guided feature decoupling module in task $b-1$ for clarity. (b) The process of constructing augmented emotional relation graph with label disambiguation in task $b$.
• Figure 3: Diagram of relation-based knowledge distillation with two teachers in the process of training task $b$. Each discrete emotion category represents a point in the affective space formed by Arousal and Valence.
• Figure 4: Comparison results (mAP) on three datasets used in our experiment under different protocols against compared CIL methods.
• Figure 5: t-SNE on the emotion embeddings learning by AESL.