MELLM: A Flow-Guided Large Language Model for Micro-Expression Understanding

TL;DR

The paper tackles the challenge of micro-expression understanding (MEU) by bridging subtle, frame-level facial dynamics with high-level emotional inference. It introduces MEFlowDataset and MEFlowNet to accurately capture micro-motion, and the Flow-Guided Micro-Expression Understanding (FGMU) paradigm to convert optical-flow signals into structured prompts for an LLM, enabling interpretable ME analysis. An MEU-focused instruction-tuning dataset (MEU-Instruct) is built and MELLM, a LoRA-finetuned Qwen model, is trained to map motion prompts to ME interpretations with explicit reasoning. Across synthetic and cross-dataset benchmarks, MELLM achieves state-of-the-art generalization and provides human-readable explanations, highlighting practical potential for robust, interpretable affective computing.

Abstract

Micro-expressions (MEs), brief and low-intensity facial movements revealing concealed emotions, are crucial for affective computing. Despite notable progress in ME recognition, existing methods are largely confined to discrete emotion classification, lacking the capacity for comprehensive ME Understanding (MEU), particularly in interpreting subtle facial dynamics and underlying emotional cues. While Multimodal Large Language Models (MLLMs) offer potential for MEU with their advanced reasoning abilities, they still struggle to perceive such subtle facial affective behaviors. To bridge this gap, we propose a ME Large Language Model (MELLM) that integrates optical flow-based sensitivity to subtle facial motions with the powerful inference ability of LLMs. Specifically, an iterative, warping-based optical-flow estimator, named MEFlowNet, is introduced to precisely capture facial micro-movements. For its training and evaluation, we construct MEFlowDataset, a large-scale optical-flow dataset with 54,611 onset-apex image pairs spanning diverse identities and subtle facial motions. Subsequently, we design a Flow-Guided Micro-Expression Understanding paradigm. Under this framework, the optical flow signals extracted by MEFlowNet are leveraged to build MEU-Instruct, an instruction-tuning dataset for MEU. MELLM is then fine-tuned on MEU-Instruct, enabling it to translate subtle motion patterns into human-readable descriptions and generate corresponding emotional inferences. Experiments demonstrate that MEFlowNet significantly outperforms existing optical flow methods in facial and ME-flow estimation, while MELLM achieves state-of-the-art accuracy and generalization across multiple ME benchmarks. To the best of our knowledge, this work presents two key contributions: MEFlowNet as the first dedicated ME flow estimator, and MELLM as the first LLM tailored for MEU.

Figures (8)

• Figure 1: Limitations of optical-flow algorithms and MLLM for MEU. (a) Optical-flow estimates produced by various methods. (b) Examples of several state-of-the-art MLLMs on MEU tasks.
• Figure 2: An overview of our proposed approach compared to other methods. (a) Traditional MER models act as black boxes, providing uninterpretable results. (b) MLLMs struggle to perceive the fine-grained dynamic changes inherent in MEs. (c) MLLMs are unable to directly process optical flow data or its visualizations. (d) Our approach converts optical flow from key ROIs into a structured Motion Prompt, enabling the LLM to effectively reason about the underlying ME.
• Figure 3: The construction pipeline of the MEFlowDataset. First, FLAME parameters ($\beta, \theta_j, \theta_h, \varphi , b$) and the UV-texture are extracted from the input facial image using the Teaser Encoder and FreeUV, respectively. These parameters are then scaled and fed into the FLAME model to generate three distinct meshes: the onset $V_{\text{onset}}$, head pose $V_{\text{head}}$, and apex $V_{\text{apex}}$ facial meshes. Finally, synthetic facial images and their corresponding ground truth head $f^{\text{gt}}_{\text{head}}$ and facial $f^{\text{gt}}_{\text{facial}}$ flows are generated using Blender.
• Figure 4: The architecture of MEFlowNet, which employs a two-stage training method. The predicted ME flow $f^{\text{pr}}_{\text{expr}}$ is derived by subtracting the head flow $f^{\text{pr}}_{\text{head}}$ from the facial flow $f^{\text{pr}}_{\text{facial}}$.
• Figure 5: The overall architecture of the MELLM.