Multi-Track Multimodal Learning on iMiGUE: Micro-Gesture and Emotion Recognition

TL;DR

The paper tackles fine-grained micro-gesture recognition and behavior-based emotion prediction from video on the iMiGUE dataset. It introduces two multimodal frameworks: a micro-gesture model that fuses RGB with 3D pose via Cross-Modal Token Fusion and a Memory-Powered Refinement module (with $\,\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{c}} + \alpha \mathcal{L}_{\text{p}}$) and a dual-stream emotion model that integrates contextual and facial cues using Iterative InterFusion fusion. Key contributions include token-level cross-modal fusion, a memory-driven refinement mechanism for gestures, and a gated inter-modal fusion strategy for emotion recognition, evaluated on an identity-free dataset. The methods achieve competitive results, including second place on the MiGA 2025 Challenge Track 3, highlighting the value of fine-grained spatio-temporal modeling and robust multimodal interaction for subtle human behavior understanding. This work demonstrates practical impact for systems requiring accurate interpretation of micro-gestures and affect in real-world settings, without relying on audio or transcripts.

Abstract

Micro-gesture recognition and behavior-based emotion prediction are both highly challenging tasks that require modeling subtle, fine-grained human behaviors, primarily leveraging video and skeletal pose data. In this work, we present two multimodal frameworks designed to tackle both problems on the iMiGUE dataset. For micro-gesture classification, we explore the complementary strengths of RGB and 3D pose-based representations to capture nuanced spatio-temporal patterns. To comprehensively represent gestures, video, and skeletal embeddings are extracted using MViTv2-S and 2s-AGCN, respectively. Then, they are integrated through a Cross-Modal Token Fusion module to combine spatial and pose information. For emotion recognition, our framework extends to behavior-based emotion prediction, a binary classification task identifying emotional states based on visual cues. We leverage facial and contextual embeddings extracted using SwinFace and MViTv2-S models and fuse them through an InterFusion module designed to capture emotional expressions and body gestures. Experiments conducted on the iMiGUE dataset, within the scope of the MiGA 2025 Challenge, demonstrate the robust performance and accuracy of our method in the behavior-based emotion prediction task, where our approach secured 2nd place.

Figures (2)

• Figure 1: Detailed architecture diagrams for microgesture classification model.
• Figure 2: (a) Architecture of the proposed multimodal architecture for emotion recognition from video and facial features. (b) The structure of the InterFusion module. (c) The $\alpha$-Gate mechanism for information aggregation from two modalities.