FEALLM: Advancing Facial Emotion Analysis in Multimodal Large Language Models with Emotional Synergy and Reasoning

TL;DR

FEALLM tackles core gaps in facial emotion analysis by creating a dedicated FEA Instruction Dataset that links facial action units (AUs) to expressions and enables causal reasoning, and by introducing FEABench to jointly evaluate FER and AUD. The core model FEALLM fuses local facial cues and low-level visual features through a Local Clue Aggregator and Multi-perspective Projector while keeping the large language model frozen and training lightweight adapters. Experiments show strong zero-shot generalization to RAF-DB, AffectNet, BP4D, and DISFA, and FEABench assessments demonstrate superior FER and AU detection over strong baselines. This work advances interpretable, generalizable FEA in multimodal LLMs, with potential for broader multimedia affective computing applications.

Abstract

Facial Emotion Analysis (FEA) plays a crucial role in visual affective computing, aiming to infer a person's emotional state based on facial data. Scientifically, facial expressions (FEs) result from the coordinated movement of facial muscles, which can be decomposed into specific action units (AUs) that provide detailed emotional insights. However, traditional methods often struggle with limited interpretability, constrained generalization and reasoning abilities. Recently, Multimodal Large Language Models (MLLMs) have shown exceptional performance in various visual tasks, while they still face significant challenges in FEA due to the lack of specialized datasets and their inability to capture the intricate relationships between FEs and AUs. To address these issues, we introduce a novel FEA Instruction Dataset that provides accurate and aligned FE and AU descriptions and establishes causal reasoning relationships between them, followed by constructing a new benchmark, FEABench. Moreover, we propose FEALLM, a novel MLLM architecture designed to capture more detailed facial information, enhancing its capability in FEA tasks. Our model demonstrates strong performance on FEABench and impressive generalization capability through zero-shot evaluation on various datasets, including RAF-DB, AffectNet, BP4D, and DISFA, showcasing its robustness and effectiveness in FEA tasks. The dataset and code will be available at https://github.com/953206211/FEALLM.

Figures (7)

• Figure 1: Illustration of the innovativeness of the proposed FEALLM. (a) shows that traditional methods using binary labels result in a lack of interpretability and generalizability. (b) shows that existing general-purpose MLLMs typically exhibit poor facial emotion analysis capabilities. (c) shows that certain MLLMs designed for FER fail to link abstract emotions with facial movements, limiting the model's performance. (d) shows that our FEALLM is endowed with strong facial emotion perception and reasoning capabilities, owing to incorporating comprehensive FEA instruction tuning and capturing more detailed facial characteristics.
• Figure 2: (a) The pipeline for generating FEA instruction data, and the distribution of (b) AUs and (c) FEs in the FEA dataset.
• Figure 3: An example of the FEA Instruction Dataset. The face image and its corresponding labels are provided as input to GPT-4o to generate a structured description, which is then split and reorganized into three types of instructions: emotion summary, facial movement description, and emotion reasoning description.
• Figure 4: (a) The overview of the FEALLM architecture. (b) The Multi-perspective Projector integrates shallow features from the visual encoder and local region features into deep features to enhance the perception of low-level facial details.
• Figure 5: Ablation study on cropping types in the Local Clue Aggregator module.