MER-CLIP: AU-Guided Vision-Language Alignment for Micro-Expression Recognition

TL;DR

MER-CLIP tackles micro-expression recognition by combining AU-described textual prompts with a video encoder through AU-guided cross-modal alignment in a CLIP framework. It introduces LocalStaticFaceMix to augment ME data without destroying motion cues, and an Emotion Inference Transformer head to bridge motion features with emotion semantics. Comprehensive experiments across SAMM, CASME II, CAS(ME)^3, and DFME show state-of-the-art UF1/UAR across 3-, 4-, and 7-class tasks, with strong performance especially on larger datasets. The approach emphasizes interpretability via Grad-CAM and t-SNE analyses, and demonstrates the value of anatomically grounded cross-modal supervision for robust MER. Potential extensions include semi-supervised AU labeling and learnable prompts to further improve generalization.

Abstract

As a critical psychological stress response, micro-expressions (MEs) are fleeting and subtle facial movements revealing genuine emotions. Automatic ME recognition (MER) holds valuable applications in fields such as criminal investigation and psychological diagnosis. The Facial Action Coding System (FACS) encodes expressions by identifying activations of specific facial action units (AUs), serving as a key reference for ME analysis. However, current MER methods typically limit AU utilization to defining regions of interest (ROIs) or relying on specific prior knowledge, often resulting in limited performance and poor generalization. To address this, we integrate the CLIP model's powerful cross-modal semantic alignment capability into MER and propose a novel approach namely MER-CLIP. Specifically, we convert AU labels into detailed textual descriptions of facial muscle movements, guiding fine-grained spatiotemporal ME learning by aligning visual dynamics and textual AU-based representations. Additionally, we introduce an Emotion Inference Module to capture the nuanced relationships between ME patterns and emotions with higher-level semantic understanding. To mitigate overfitting caused by the scarcity of ME data, we put forward LocalStaticFaceMix, an effective data augmentation strategy blending facial images to enhance facial diversity while preserving critical ME features. Finally, comprehensive experiments on four benchmark ME datasets confirm the superiority of MER-CLIP. Notably, UF1 scores on CAS(ME)3 reach 0.7832, 0.6544, and 0.4997 for 3-, 4-, and 7-class classification tasks, significantly outperforming previous methods.

Figures (7)

• Figure 1: Examples of a MaE (a) and a ME (b) from the same person in the DFME database zhao2023dfme, both reflecting the 'happiness' emotional state. To facilitate observation, we have annotated their activated action units related to specific facial muscle movements based on FACS. The MaE includes a combination of Cheek Raiser (AU6), Lip Corner Puller (AU12), and Lips Part (AU25). The ME only includes slight AU6 and AU12.
• Figure 2: We illustrate the detailed architecture of MER-CLIP. Before being fed into the video encoder, the input video undergoes a series of augmentations, including the basic transformations such as color jitter and flipping, Augmix, and our proposed LocalStaticFaceMix. The augmented video is then passed through the Video Motion Encoding to extract the spatiotemporal ME movement features. These features are subsequently sent into two main components: the AU-Guided Cross-Modal Alignment Module and the Emotion Inference Module to perform fine-grained cross-modal alignment and emotion learning.
• Figure 3: The detailed illustration of proposed LocalStaticFaceMix.
• Figure 4: The detailed flowchart of the Video Motion Encoding Process.
• Figure 5: Confusion matrices of MER tasks on CASME II, SAMM, CAS(ME$)^3$ and DFME datasets.