Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Anchoring Emotions in Text: Robust Multimodal Fusion for Mimicry Intensity Estimation

Lingsi Zhu, Yuefeng Zou, Yunxiang Zhang, Naixiang Zheng, Guoyuan Wang, Jun Yu, Jiaen Liang, Wei Huang, Shengping Liu, Ximin Zheng

Abstract

Estimating Emotional Mimicry Intensity (EMI) in naturalistic environments is a critical yet challenging task in affective computing. The primary difficulty lies in effectively modeling the complex, nonlinear temporal dynamics across highly heterogeneous modalities, especially when physical signals are corrupted or missing. To tackle this, we propose TAEMI (Text-Anchored Emotional Mimicry Intensity estimation), a novel multimodal framework designed for the 10th ABAW Competition. Motivated by the observation that continuous visual and acoustic signals are highly susceptible to transient environmental noise, we break the traditional symmetric fusion paradigm. Instead, we leverage textual transcript--which inherently encode a stable, time-independent semantic prior--as central anchors. Specifically, we introduce a Text-Anchored Dual Cross-Attention mechanism that utilizes these robust textual queries to actively filter out frame-level redundancies and align the noisy physical streams. Furthermore, to prevent catastrophic performance degradation caused by inevitably missing data in unconstrained real-world scenarios, we integrate Learnable Missing-Modality Tokens and a Modality Dropout strategy during training. Extensive experiments on the Hume-Vidmimic2 dataset demonstrate that TAEMI effectively captures fine-grained emotional variations and maintains robust predictive resilience under imperfect conditions. Our framework achieves a state-of-the-art mean Pearson correlation coefficient across six continuous emotional dimensions, significantly outperforming existing baseline methods.

Anchoring Emotions in Text: Robust Multimodal Fusion for Mimicry Intensity Estimation

Abstract

Estimating Emotional Mimicry Intensity (EMI) in naturalistic environments is a critical yet challenging task in affective computing. The primary difficulty lies in effectively modeling the complex, nonlinear temporal dynamics across highly heterogeneous modalities, especially when physical signals are corrupted or missing. To tackle this, we propose TAEMI (Text-Anchored Emotional Mimicry Intensity estimation), a novel multimodal framework designed for the 10th ABAW Competition. Motivated by the observation that continuous visual and acoustic signals are highly susceptible to transient environmental noise, we break the traditional symmetric fusion paradigm. Instead, we leverage textual transcript--which inherently encode a stable, time-independent semantic prior--as central anchors. Specifically, we introduce a Text-Anchored Dual Cross-Attention mechanism that utilizes these robust textual queries to actively filter out frame-level redundancies and align the noisy physical streams. Furthermore, to prevent catastrophic performance degradation caused by inevitably missing data in unconstrained real-world scenarios, we integrate Learnable Missing-Modality Tokens and a Modality Dropout strategy during training. Extensive experiments on the Hume-Vidmimic2 dataset demonstrate that TAEMI effectively captures fine-grained emotional variations and maintains robust predictive resilience under imperfect conditions. Our framework achieves a state-of-the-art mean Pearson correlation coefficient across six continuous emotional dimensions, significantly outperforming existing baseline methods.
Paper Structure (16 sections, 7 equations, 1 figure, 4 tables)

This paper contains 16 sections, 7 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Emotion Recognition Techniques
  4. Affective Computing and Emotional Mimicry Intensity
  5. Methodology
  6. Problem Formulation
  7. Data Processing and Sequence Construction
  8. Text-Anchored Dual Cross-Attention Fusion
  9. Robustness Mechanisms
  10. Optimization Strategy
  11. Experiment
  12. Setup
  13. Main Results
  14. Ablation Study
  15. Conclusion
  16. ...and 1 more sections

Figures (1)

  • Figure 1: The overall architecture of TAEMI. The framework processes temporally aligned audio, visual, and textual modalities. Following modality-specific feature extraction and linear alignment, the temporal representations are fed into a Text-Anchored Dual Cross-Attention module. Here, the textual features serve as queries to aggregate pertinent audio and visual contexts. The resulting fused multimodal representation is then processed by a Multi-Layer Perceptron (MLP) to predict the continuous emotion intensity vector. During training, learnable missing-modality tokens and modality dropout are strategically incorporated to enhance the model's robustness.