Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Sentiment Analysis based on Video and Audio Inputs

Antonio Fernandez, Suzan Awinat

TL;DR

The paper tackles multimodal sentiment analysis by combining audio (CREMA-D) and video (RAVDESS) inputs using fine-tuned wav2vec2-large for audio and vivit-b-16x2-kinetics400 for video. It explores several fusion strategies—averaging, confidence-based thresholds, dynamic weighting, and rule-based logic—to fuse per-modality emotion probabilities and improve accuracy. Results show that joint fusion can surpass individual modalities, with averaging and rule-based methods performing best when modality accuracies are similar, though large performance gaps can reduce gains. The study discusses limitations such as dataset bias and controlled conditions, and outlines future directions including adding an NLP-based modality and potential real-time robotic applications, while noting regulatory considerations under the EU AI Act.

Abstract

Despite the abundance of current researches working on the sentiment analysis from videos and audios, finding the best model that gives the highest accuracy rate is still considered a challenge for researchers in this field. The main objective of this paper is to prove the usability of emotion recognition models that take video and audio inputs. The datasets used to train the models are the CREMA-D dataset for audio and the RAVDESS dataset for video. The fine-tuned models that been used are: Facebook/wav2vec2-large for audio and the Google/vivit-b-16x2-kinetics400 for video. The avarage of the probabilities for each emotion generated by the two previous models is utilized in the decision making framework. After disparity in the results, if one of the models gets much higher accuracy, another test framework is created. The methods used are the Weighted Average method, the Confidence Level Threshold method, the Dynamic Weighting Based on Confidence method, and the Rule-Based Logic method. This limited approach gives encouraging results that make future research into these methods viable.

Multimodal Sentiment Analysis based on Video and Audio Inputs

TL;DR

The paper tackles multimodal sentiment analysis by combining audio (CREMA-D) and video (RAVDESS) inputs using fine-tuned wav2vec2-large for audio and vivit-b-16x2-kinetics400 for video. It explores several fusion strategies—averaging, confidence-based thresholds, dynamic weighting, and rule-based logic—to fuse per-modality emotion probabilities and improve accuracy. Results show that joint fusion can surpass individual modalities, with averaging and rule-based methods performing best when modality accuracies are similar, though large performance gaps can reduce gains. The study discusses limitations such as dataset bias and controlled conditions, and outlines future directions including adding an NLP-based modality and potential real-time robotic applications, while noting regulatory considerations under the EU AI Act.

Abstract

Despite the abundance of current researches working on the sentiment analysis from videos and audios, finding the best model that gives the highest accuracy rate is still considered a challenge for researchers in this field. The main objective of this paper is to prove the usability of emotion recognition models that take video and audio inputs. The datasets used to train the models are the CREMA-D dataset for audio and the RAVDESS dataset for video. The fine-tuned models that been used are: Facebook/wav2vec2-large for audio and the Google/vivit-b-16x2-kinetics400 for video. The avarage of the probabilities for each emotion generated by the two previous models is utilized in the decision making framework. After disparity in the results, if one of the models gets much higher accuracy, another test framework is created. The methods used are the Weighted Average method, the Confidence Level Threshold method, the Dynamic Weighting Based on Confidence method, and the Rule-Based Logic method. This limited approach gives encouraging results that make future research into these methods viable.

Paper Structure

This paper contains 10 sections, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodologies and Preliminary Analyses
  3. Audio Classification Dataset
  4. Video Classification Dataset
  5. Models and Framework
  6. Results
  7. Audio Model Training
  8. Video Model Training
  9. Framework Scores
  10. Discussion and Conclusion

Figures (6)

  • Figure 1: The entire process of our module
  • Figure 2: Accuracy of Different Models Using Averaging Method
  • Figure 3: Accuracy of Different Models Using Weighted Average Method
  • Figure 4: Accuracy of Different Models Using Confidence Level Threshold on the Video Model Method
  • Figure 5: Accuracy of Different Models Using Dynamic Weighting Based on Confidence Method
  • ...and 1 more figures