Pairwise Discernment of AffectNet Expressions with ArcFace

TL;DR

This work investigates transferring knowledge from ImageNet-trained backbones and ArcFace to Facial Emotion Recognition on AffectNet, emphasizing the impact of data imbalance and exploring pairwise discrimination as a remedy. The study finds modest gains from domain transfer but reveals substantial challenges posed by minority-class scarcity and annotation subjectivity; a pairwise discernment strategy yields improvements for some emotion pairs yet does not universally outperform a single-model baseline. Key contributions include a systematic pairing approach to FER, an analysis of loss functions and architectures in a transfer-learning FER setting, and evidence that pairwise learning can help isolate discriminative features for minority emotions. The results underscore the need for more balanced, well-annotated data and point toward future directions like specialized pairwise models and probabilistic emotion representations.

Abstract

This study takes a preliminary step toward teaching computers to recognize human emotions through Facial Emotion Recognition (FER). Transfer learning is applied using ResNeXt, EfficientNet models, and an ArcFace model originally trained on the facial verification task, leveraging the AffectNet database, a collection of human face images annotated with corresponding emotions. The findings highlight the value of congruent domain transfer learning, the challenges posed by imbalanced datasets in learning facial emotion patterns, and the effectiveness of pairwise learning in addressing class imbalances to enhance model performance on the FER task.