Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MicroEmo: Time-Sensitive Multimodal Emotion Recognition with Micro-Expression Dynamics in Video Dialogues

Liyun Zhang

TL;DR

MicroEmo tackles time-sensitive multimodal emotion recognition by capturing micro-expression dynamics and utterance-aware video context in open-vocabulary settings. It introduces a global-local attention visual encoder and an utterance-aware video Q-Former to produce multi-scale visual tokens that align with audio and transcribed speech, subsequently processed by a large language model under a two-stage training regime with LoRA fine-tuning. Experiments on the EMER/EMER-Multi task show MicroEmo outperforming baselines, with ablations confirming the necessity of both novel modules. The work advances explainable, open-vocabulary emotion recognition in videos and lays groundwork for extending the utterance-aware paradigm to acoustic modalities.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable multimodal emotion recognition capabilities, integrating multimodal cues from visual, acoustic, and linguistic contexts in the video to recognize human emotional states. However, existing methods ignore capturing local facial features of temporal dynamics of micro-expressions and do not leverage the contextual dependencies of the utterance-aware temporal segments in the video, thereby limiting their expected effectiveness to a certain extent. In this work, we propose MicroEmo, a time-sensitive MLLM aimed at directing attention to the local facial micro-expression dynamics and the contextual dependencies of utterance-aware video clips. Our model incorporates two key architectural contributions: (1) a global-local attention visual encoder that integrates global frame-level timestamp-bound image features with local facial features of temporal dynamics of micro-expressions; (2) an utterance-aware video Q-Former that captures multi-scale and contextual dependencies by generating visual token sequences for each utterance segment and for the entire video then combining them. Preliminary qualitative experiments demonstrate that in a new Explainable Multimodal Emotion Recognition (EMER) task that exploits multi-modal and multi-faceted clues to predict emotions in an open-vocabulary (OV) manner, MicroEmo demonstrates its effectiveness compared with the latest methods.

MicroEmo: Time-Sensitive Multimodal Emotion Recognition with Micro-Expression Dynamics in Video Dialogues

TL;DR

MicroEmo tackles time-sensitive multimodal emotion recognition by capturing micro-expression dynamics and utterance-aware video context in open-vocabulary settings. It introduces a global-local attention visual encoder and an utterance-aware video Q-Former to produce multi-scale visual tokens that align with audio and transcribed speech, subsequently processed by a large language model under a two-stage training regime with LoRA fine-tuning. Experiments on the EMER/EMER-Multi task show MicroEmo outperforming baselines, with ablations confirming the necessity of both novel modules. The work advances explainable, open-vocabulary emotion recognition in videos and lays groundwork for extending the utterance-aware paradigm to acoustic modalities.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable multimodal emotion recognition capabilities, integrating multimodal cues from visual, acoustic, and linguistic contexts in the video to recognize human emotional states. However, existing methods ignore capturing local facial features of temporal dynamics of micro-expressions and do not leverage the contextual dependencies of the utterance-aware temporal segments in the video, thereby limiting their expected effectiveness to a certain extent. In this work, we propose MicroEmo, a time-sensitive MLLM aimed at directing attention to the local facial micro-expression dynamics and the contextual dependencies of utterance-aware video clips. Our model incorporates two key architectural contributions: (1) a global-local attention visual encoder that integrates global frame-level timestamp-bound image features with local facial features of temporal dynamics of micro-expressions; (2) an utterance-aware video Q-Former that captures multi-scale and contextual dependencies by generating visual token sequences for each utterance segment and for the entire video then combining them. Preliminary qualitative experiments demonstrate that in a new Explainable Multimodal Emotion Recognition (EMER) task that exploits multi-modal and multi-faceted clues to predict emotions in an open-vocabulary (OV) manner, MicroEmo demonstrates its effectiveness compared with the latest methods.
Paper Structure (13 sections, 1 equation, 1 figure, 2 tables)

This paper contains 13 sections, 1 equation, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Methodology
  4. Overview
  5. Global-Local Attention Visual Encoder
  6. Utterance-Aware Video Q-Former
  7. Large Language Model
  8. Experiments
  9. Implementation Details
  10. Experiment Setup
  11. Main Results
  12. Ablation Study
  13. Conclusion

Figures (1)

  • Figure 1: The overall architecture of MicroEmo. Input a sequence of video frames along with their timestamps, (a) Global-local attention visual encoder first extracts and integrates global frame-level timestamp-bound image features with local facial features of temporal dynamics of micro-expressions. Then (b) utterance-aware video Q-Former combines utterance-level timestamps to generate visual tokens for each utterance segment via an utterance-aware sliding window and combines them with whole tokens extracted by a global video Q-Former to produce multi-scale fused visual tokens. Audio is also passed through a pre-trained encoder and whole video Q-Former to obtain audio tokens. Finally, these multimodal tokens are concatenated with transcribed speech and instructions, which are then fed into a (c) Large Language Model to generate responses.