Cross-Task Inconsistency Based Active Learning (CTIAL) for Emotion Recognition

TL;DR

The paper tackles the labeling bottleneck in emotion recognition by introducing Cross-Task Inconsistency Based Active Learning (CTIAL), which leverages knowledge from both categorical and dimensional emotion representations. Affective norms from the NRC Lexicon map categorical predictions into the dimensional space, enabling a Cross-Task Inconsistency (CTI) metric to guide sample selection across tasks. CTIAL combines CTI with within-task uncertainty or diversity measures and employs domain adaptation (TCA/BDA) to handle cross-corpus shifts, achieving improved sample efficiency and transfer performance. Experiments on IEMOCAP, MELD, and VAM across within- and cross-corpus settings demonstrate that CTIAL enhances labeling efficiency, improves cross-task transfer, and is the first to exploit affective-norm knowledge across tasks for active learning in emotion recognition.

Abstract

Emotion recognition is a critical component of affective computing. Training accurate machine learning models for emotion recognition typically requires a large amount of labeled data. Due to the subtleness and complexity of emotions, multiple evaluators are usually needed for each affective sample to obtain its ground-truth label, which is expensive. To save the labeling cost, this paper proposes an inconsistency-based active learning approach for cross-task transfer between emotion classification and estimation. Affective norms are utilized as prior knowledge to connect the label spaces of categorical and dimensional emotions. Then, the prediction inconsistency on the two tasks for the unlabeled samples is used to guide sample selection in active learning for the target task. Experiments on within-corpus and cross-corpus transfers demonstrated that cross-task inconsistency could be a very valuable metric in active learning. To our knowledge, this is the first work that utilizes prior knowledge on affective norms and data in a different task to facilitate active learning for a new task, even the two tasks are from different datasets.

Figures (8)

• Figure 1: Within-task AL for DEE, and CTIAL for cross-task transfer from CEC to DEE.
• Figure 2: Flowchart for computing the CTI.
• Figure 3: Average BCAs of different sample selection approaches in cross-task transfer from DEE to CEC. (a) Within-corpus transfer from DEE on IEMOCAP to CEC on IEMOCAP; and, (b) cross-corpus transfer from DEE on VAM to CEC on IEMOCAP. $K$ is the number of samples to be queried in addition to the initial labeled ones.
• Figure 4: Statistical significance of the performance improvements of Ent-CTIAL and LC-CTIAL over the other approaches. (a) Within-corpus transfer from DEE on IEMOCAP to CEC on IEMOCAP; and, (b) cross-corpus transfer from DEE on VAM to CEC on IEMOCAP. $K$ is the number of samples to be queried in addition to the initial labeled ones. The vertical axis denotes the approaches in comparison with Ent-CTIAL or LC-CTIAL in Wilcoxon signed-rank tests. The red and green markers were placed at where the adjusted $p$-values were smaller than 0.05.
• Figure 5: Average RMSEs and CCs in valence, arousal and dominance estimation on IEMOCAP in within-corpus transfer (DEE on IEMOCAP to DEE on IEMOCAP; red line), direct cross-corpus transfer (DEE on VAM to DEE on IEMOCAP; blue line), and cross-corpus transfer using TCA (DEE on VAM to DEE on IEMOCAP; black curve). $d$ is the feature dimensionality. The markers on the red and blue lines mean that the feature dimensionality after principal component analysis was 46 and 40, respectively.