MotivNet: Evolving Meta-Sapiens into an Emotionally Intelligent Foundation Model

TL;DR

Facial expression recognition (FER) models often fail to generalize to real-world, unconstrained data due to dataset bias and domain shift. The authors propose MotivNet, a transfer-learning FER model that uses the Meta-Sapiens human-vision foundation model as a backbone and treats MotivNet as a Sapiens downstream task; MotivNet is trained on a uniformly sampled AffectNet subset to ensure data similarity with the Sapiens pretraining regime. They introduce three viability criteria—architecture similarity to Sapiens, data similarity to Sapiens pretraining data, and competitive benchmark performance—to validate the approach. Empirically, MotivNet achieves competitive weighted recall ($WAR$) and Top-2 accuracy across multiple datasets (JAFFE, CK+, FER-2013, AffectNet) without cross-domain training, approaching or matching state-of-the-art cross-domain FER models. This work supports leveraging large-scale, human-centric foundation models to enable robust FER in the wild and invites broader adoption of foundation-model strategies for domain-specific perception tasks.

Abstract

In this paper, we introduce MotivNet, a generalizable facial emotion recognition model for robust real-world application. Current state-of-the-art FER models tend to have weak generalization when tested on diverse data, leading to deteriorated performance in the real world and hindering FER as a research domain. Though researchers have proposed complex architectures to address this generalization issue, they require training cross-domain to obtain generalizable results, which is inherently contradictory for real-world application. Our model, MotivNet, achieves competitive performance across datasets without cross-domain training by using Meta-Sapiens as a backbone. Sapiens is a human vision foundational model with state-of-the-art generalization in the real world through large-scale pretraining of a Masked Autoencoder. We propose MotivNet as an additional downstream task for Sapiens and define three criteria to evaluate MotivNet's viability as a Sapiens task: benchmark performance, model similarity, and data similarity. Throughout this paper, we describe the components of MotivNet, our training approach, and our results showing MotivNet is generalizable across domains. We demonstrate that MotivNet can be benchmarked against existing SOTA models and meets the listed criteria, validating MotivNet as a Sapiens downstream task, and making FER more incentivizing for in-the-wild application. The code is available at https://github.com/OSUPCVLab/EmotionFromFaceImages.

Figures (4)

• Figure 1: An example of FER class imbalance by graphing the frequency of each label in the AffectNet dataset. khirodkar2024sapiensfoundationhumanvision
• Figure 2: Sapiens MAE reconstruction. Each triplet shows ground truth on the left, the mask in the middle, and the reconstruction on the right. The bottom shows different triplets under a 75% and 95% mask ratio. Adapted from Sapiens: Foundation for Human Vision Modelskhirodkar2024sapiensfoundationhumanvision
• Figure 3: Examples of the eight types of labeled facial images in AffectNet. Adapted from AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the WildMollahosseini_2019.
• Figure 4: A diagram of MotivNet's encoder-decoder architecture. The encoder is Sapiens' network consisting of a patch embedding, dropout, and 40 transformer-encoder layers. The decoder is ML-Decoder's network consisting of a cross attention mechanism with non-learnable group queries feeding into a fully connected network and then a novel group fully connected pooling. Adapted from Sapiens: Foundation for Human Vision Models and ML-Decoder: Scalable and Versatile Classification Headkhirodkar2024sapiensfoundationhumanvisionridnik2021mldecoderscalableversatileclassification.