Compound Expression Recognition via Multi Model Ensemble for the ABAW7 Challenge

TL;DR

This work tackles Compound Expression Recognition (CER) by integrating local and global facial cues through a late-fusion ensemble of encoders based on ResNet (CNN), Vision Transformer, and POSTER architectures. Features from PosterV2 (768-d) and a CNN backbone (2048-d) are concatenated and fed into an MLP to predict seven compound expressions, trained on a unified RAF-DB/AffectNet-derived dataset and evaluated on RAF-DB and C-EXPR-DB. The approach demonstrates strong performance, with ViT showing strong per-expression accuracy and the ensemble providing robust gains across expressions, including notable improvements on difficult cases such as Sadly Surprised. This method advances practical CER in the wild by leveraging complementary representations and large-scale pretraining, enabling improved affective understanding for human-computer interaction and related ABAW challenges.

Abstract

Compound Expression Recognition (CER) is vital for effective interpersonal interactions. Human emotional expressions are inherently complex due to the presence of compound expressions, requiring the consideration of both local and global facial cues for accurate judgment. In this paper, we propose an ensemble learning-based solution to address this complexity. Our approach involves training three distinct expression classification models using convolutional networks, Vision Transformers, and multiscale local attention networks. By employing late fusion for model ensemble, we combine the outputs of these models to predict the final results. Our method demonstrates high accuracy on the RAF-DB datasets and is capable of recognizing expressions in certain portions of the C-EXPR-DB through zero-shot learning.

Figures (1)

• Figure 1: The image and its two distinct enhanced versions are processed through the PosterV2 model and the ResNet50 model to obtain 768-dimensional and 2048-dimensional features, respectively. These features are then concatenated as input for the subsequent stage. The features extracted from the original image are separately input into two Multi-Layer Perceptron (MLP) to obtain basic and compound expression predictions. The basic expression predictions are learned through the loss function $\mathcal{L}_{basic}$. Concurrently, the basic expression predictions are combined with the compound expression predictions and learned through another loss function $\mathcal{L}_{ce}$. Additionally, to better adapt the model to various datasets, we employ contrastive learning as an optimization technique by the loss function $\mathcal{L}_{CL}$.