GCM-Net: Graph-enhanced Cross-Modal Infusion with a Metaheuristic-Driven Network for Video Sentiment and Emotion Analysis

TL;DR

This paper presents a novel framework that leverages the multi-modal contextual information from utterances and applies metaheuristic algorithms to learn the contributing features for utterance-level sentiment and emotion prediction and demonstrates the efficacy of this approach on three prominent multi-modal benchmark datasets.

Abstract

Sentiment analysis and emotion recognition in videos are challenging tasks, given the diversity and complexity of the information conveyed in different modalities. Developing a highly competent framework that effectively addresses the distinct characteristics across various modalities is a primary concern in this domain. Previous studies on combined multimodal sentiment and emotion analysis often overlooked effective fusion for modality integration, intermodal contextual congruity, optimizing concatenated feature spaces, leading to suboptimal architecture. This paper presents a novel framework that leverages the multi-modal contextual information from utterances and applies metaheuristic algorithms to learn the contributing features for utterance-level sentiment and emotion prediction. Our Graph-enhanced Cross-Modal Infusion with a Metaheuristic-Driven Network (GCM-Net) integrates graph sampling and aggregation to recalibrate the modality features for video sentiment and emotion prediction. GCM-Net includes a cross-modal attention module determining intermodal interactions and utterance relevance. A harmonic optimization module employing a metaheuristic algorithm combines attended features, allowing for handling both single and multi-utterance inputs. To show the effectiveness of our approach, we have conducted extensive evaluations on three prominent multi-modal benchmark datasets, CMU MOSI, CMU MOSEI, and IEMOCAP. The experimental results demonstrate the efficacy of our proposed approach, showcasing accuracies of 91.56% and 86.95% for sentiment analysis on MOSI and MOSEI datasets. We have performed emotion analysis for the IEMOCAP dataset procuring an accuracy of 85.66% which signifies substantial performance enhancements over existing methods.

Figures (9)

• Figure 1: Generic Methodology Structure for Multimodal Sentiment Analysis and Emotion Prediction
• Figure 2: This figure illustrates our Graph-enhanced Cross-Modal Infusion (GCM-Net) architecture for video sentiment and emotion analysis. Modality-specific features are recalibrated via a graph-based approach, followed by dynamic weighting through ICIM. A harmonic optimization algorithm then selects optimal feature subsets for the ConvXGB classifier, achieving efficient and accurate prediction.
• Figure 3: Illustration of Graph-based Feature Recalibration and Enrichment (FRE) which begins by constructing an adjacency matrix, linking utterances exceeding a similarity threshold. Neighboring features are then aggregated and sampled for node optimization
• Figure 4: Intermodal Contextual Interaction Module (ICIM). This module facilitates cross-modal interaction by computing pairwise attentions between different modalities
• Figure 5: Performance of Considered Models on CMU-MOSI for Sentiment Analysis