Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Self-Attention Network with Temporal Alignment for Audio-Visual Emotion Recognition

Inyong Koo, yeeun Seong, Minseok Son, Jaehyuk Jang, Changick Kim

Abstract

Audio-visual emotion recognition (AVER) methods typically fuse utterance-level features, and even frame-level attention models seldom address the frame-rate mismatch across modalities. In this paper, we propose a Transformer-based framework focusing on the temporal alignment of multimodal features. Our design employs a multimodal self-attention encoder that simultaneously captures intra- and inter-modal dependencies within a shared feature space. To address heterogeneous sampling rates, we incorporate Temporally-aligned Rotary Position Embeddings (TaRoPE), which implicitly synchronize audio and video tokens. Furthermore, we introduce a Cross-Temporal Matching (CTM) loss that enforces consistency among temporally proximate pairs, guiding the encoder toward better alignment. Experiments on CREMA-D and RAVDESS datasets demonstrate consistent improvements over recent baselines, suggesting that explicitly addressing frame-rate mismatch helps preserve temporal cues and enhances cross-modal fusion.

Multimodal Self-Attention Network with Temporal Alignment for Audio-Visual Emotion Recognition

Abstract

Audio-visual emotion recognition (AVER) methods typically fuse utterance-level features, and even frame-level attention models seldom address the frame-rate mismatch across modalities. In this paper, we propose a Transformer-based framework focusing on the temporal alignment of multimodal features. Our design employs a multimodal self-attention encoder that simultaneously captures intra- and inter-modal dependencies within a shared feature space. To address heterogeneous sampling rates, we incorporate Temporally-aligned Rotary Position Embeddings (TaRoPE), which implicitly synchronize audio and video tokens. Furthermore, we introduce a Cross-Temporal Matching (CTM) loss that enforces consistency among temporally proximate pairs, guiding the encoder toward better alignment. Experiments on CREMA-D and RAVDESS datasets demonstrate consistent improvements over recent baselines, suggesting that explicitly addressing frame-rate mismatch helps preserve temporal cues and enhances cross-modal fusion.
Paper Structure (12 sections, 7 equations, 3 figures, 3 tables)

This paper contains 12 sections, 7 equations, 3 figures, 3 tables.

Table of Contents

  1. INTRODUCTION
  2. METHODOLOGY
  3. Architecture
  4. Temporally-aligned RoPE
  5. Cross-Temporal Matching Loss
  6. EXPERIMENTS
  7. Datasets and Evaluation Metrics
  8. Implementation Details
  9. Comparison with State-of-the-Art Methods
  10. Ablation Studies
  11. Analysis on Temporal Alignment
  12. CONCLUSION

Figures (3)

  • Figure 1: Audio and video features are first extracted by modality-specific encoders and projected into a shared embedding space. Temporally-aligned RoPE (TaRoPE) implicitly synchronizes heterogeneous audio–visual sequences within the unified Transformer encoder by adapting rotary positional embeddings to different temporal resolutions. The Cross-Temporal Matching (CTM) loss explicitly enforces temporal consistency by aligning audio–video feature pairs that are temporally proximal along a shared time axis. The final representation is pooled for emotion classification.
  • Figure 2: Illustration of differenta fusion strategies using attention mechanisms.
  • Figure 3: Effect of CTM loss on feature dynamics.