Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Fusion Method with Spatiotemporal Sequences and Relationship Learning for Valence-Arousal Estimation

Jun Yu, Gongpeng Zhao, Yongqi Wang, Zhihong Wei, Yang Zheng, Zerui Zhang, Zhongpeng Cai, Guochen Xie, Jichao Zhu, Wangyuan Zhu

TL;DR

This paper devised a comprehensive model by preprocessing video frames and audio segments to extract visual and audio features, and employed a Transformer encoder structure to learn long-range dependencies, thereby enhancing the model's performance and generalization ability.

Abstract

This paper presents our approach for the VA (Valence-Arousal) estimation task in the ABAW6 competition. We devised a comprehensive model by preprocessing video frames and audio segments to extract visual and audio features. Through the utilization of Temporal Convolutional Network (TCN) modules, we effectively captured the temporal and spatial correlations between these features. Subsequently, we employed a Transformer encoder structure to learn long-range dependencies, thereby enhancing the model's performance and generalization ability. Our method leverages a multimodal data fusion approach, integrating pre-trained audio and video backbones for feature extraction, followed by TCN-based spatiotemporal encoding and Transformer-based temporal information capture. Experimental results demonstrate the effectiveness of our approach, achieving competitive performance in VA estimation on the AffWild2 dataset.

Multimodal Fusion Method with Spatiotemporal Sequences and Relationship Learning for Valence-Arousal Estimation

TL;DR

This paper devised a comprehensive model by preprocessing video frames and audio segments to extract visual and audio features, and employed a Transformer encoder structure to learn long-range dependencies, thereby enhancing the model's performance and generalization ability.

Abstract

This paper presents our approach for the VA (Valence-Arousal) estimation task in the ABAW6 competition. We devised a comprehensive model by preprocessing video frames and audio segments to extract visual and audio features. Through the utilization of Temporal Convolutional Network (TCN) modules, we effectively captured the temporal and spatial correlations between these features. Subsequently, we employed a Transformer encoder structure to learn long-range dependencies, thereby enhancing the model's performance and generalization ability. Our method leverages a multimodal data fusion approach, integrating pre-trained audio and video backbones for feature extraction, followed by TCN-based spatiotemporal encoding and Transformer-based temporal information capture. Experimental results demonstrate the effectiveness of our approach, achieving competitive performance in VA estimation on the AffWild2 dataset.
Paper Structure (14 sections, 7 equations, 3 figures, 3 tables)

This paper contains 14 sections, 7 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed approach
  4. Feature Extraction
  5. Preprocessing
  6. Training Details
  7. LA-SE Module
  8. TCN Module
  9. Transformer Encode
  10. Experiment
  11. Dataset and Evaluation
  12. Module Ablation Experiment Results
  13. Results on Validation set
  14. Conclusion

Figures (3)

  • Figure 1: Our proposed framework for VA estimation. The model begins with preprocessing of video frames and audio segments, followed by feature extraction using pre-trained audio and video backbones. The LA-SE module enhances local image information capture and channel selection. Temporal Convolutional Network (TCN) modules capture temporal and spatial correlations in the features, while a Transformer encoder structure learns long-range dependencies.
  • Figure 2: The LANet module, where H, W and C refer to height, width and number of channels, respectively.
  • Figure 3: The SENet module, where H, W and C refer to height, width and number of channels, respectively.