EmoCLIP: A Vision-Language Method for Zero-Shot Video Facial Expression Recognition

TL;DR

This work tackles the limitation of FER models that are restricted to a fixed set of emotions by enabling zero-shot recognition of unseen expressions in video. It introduces EmoCLIP, a contrastive vision-language framework that trains video and text encoders with sample-level descriptions (captions) and leverages CLIP priors, augmented with a temporal Transformer to capture dynamics. Key contributions include using sample-level supervision for semantically rich latent spaces, representing compound emotions by averaging component embeddings, and achieving strong zero-shot performance across several datasets, as well as competitive schizophrenia symptom estimation using a linear probe on the learned video representations. The method demonstrates improved generalization over baselines like CLIP and FaRL and shows practical impact in affective computing and mental health assessment. Code availability is provided, enabling reproducibility and broader adoption.

Abstract

Facial Expression Recognition (FER) is a crucial task in affective computing, but its conventional focus on the seven basic emotions limits its applicability to the complex and expanding emotional spectrum. To address the issue of new and unseen emotions present in dynamic in-the-wild FER, we propose a novel vision-language model that utilises sample-level text descriptions (i.e. captions of the context, expressions or emotional cues) as natural language supervision, aiming to enhance the learning of rich latent representations, for zero-shot classification. To test this, we evaluate using zero-shot classification of the model trained on sample-level descriptions on four popular dynamic FER datasets. Our findings show that this approach yields significant improvements when compared to baseline methods. Specifically, for zero-shot video FER, we outperform CLIP by over 10\% in terms of Weighted Average Recall and 5\% in terms of Unweighted Average Recall on several datasets. Furthermore, we evaluate the representations obtained from the network trained using sample-level descriptions on the downstream task of mental health symptom estimation, achieving performance comparable or superior to state-of-the-art methods and strong agreement with human experts. Namely, we achieve a Pearson's Correlation Coefficient of up to 0.85 on schizophrenia symptom severity estimation, which is comparable to human experts' agreement. The code is publicly available at: https://github.com/NickyFot/EmoCLIP.

Figures (4)

• Figure 1: Overview of our method, EmoCLIP. During training (a), we use joint training to optimise the cosine similarity of video-text embedding pairs in the mini-batch. Sample-specific descriptions of the subject's facial expressions are used to train the model. During inference (b), we perform zero-shot classification using class-level descriptions for each of the emotion categories.
• Figure 2: EmoCLIP manipulates the latent space of basic emotions to create representations for compound emotions. We take the average latent representation of the components and concatenate it to the set of representations for each new compound emotion.
• Figure 3: Latent representation of the MAFW dataset using CLIP (a) and EmoCLIP (b) for each emotion category.
• Figure 4: Example frames of correctly and incorrectly classified samples from the MAFW liu_mafw_2022 dataset, for anger (top) and happiness (bottom). The sample-level descriptions are included for reference but are not used during inference.