Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Structured Prompting and LLM Ensembling for Multimodal Conversational Aspect-based Sentiment Analysis

Zhiqiang Gao, Shihao Gao, Zixing Zhang, Yihao Guo, Hongyu Chen, Jing Han

TL;DR

This work tackles multimodal conversational sentiment analysis via two MCABSA challenges: panoptic sentiment sextuple extraction and sentiment flipping analysis. It introduces a structured prompting pipeline with MSGR for robust Target–Aspect extraction, followed by Hybrid LLM Optimization Strategy (HLOS) to refine opinions, sentiments, and rationales, plus an ensemble of three LLMs for flip detection. Empirical results show substantial gains from component-wise refinements (subtask-I) and strong ensemble robustness (subtask-II), achieving 47.38% average and 74.12% Exact Match F1, respectively. The approach demonstrates the feasibility of combining multimodal captioning, stepwise extraction, and model ensembling for rich, multimodal sentiment understanding with potential impact on emotionally intelligent AI systems.

Abstract

Understanding sentiment in multimodal conversations is a complex yet crucial challenge toward building emotionally intelligent AI systems. The Multimodal Conversational Aspect-based Sentiment Analysis (MCABSA) Challenge invited participants to tackle two demanding subtasks: (1) extracting a comprehensive sentiment sextuple, including holder, target, aspect, opinion, sentiment, and rationale from multi-speaker dialogues, and (2) detecting sentiment flipping, which detects dynamic sentiment shifts and their underlying triggers. For Subtask-I, in the present paper, we designed a structured prompting pipeline that guided large language models (LLMs) to sequentially extract sentiment components with refined contextual understanding. For Subtask-II, we further leveraged the complementary strengths of three LLMs through ensembling to robustly identify sentiment transitions and their triggers. Our system achieved a 47.38% average score on Subtask-I and a 74.12% exact match F1 on Subtask-II, showing the effectiveness of step-wise refinement and ensemble strategies in rich, multimodal sentiment analysis tasks.

Structured Prompting and LLM Ensembling for Multimodal Conversational Aspect-based Sentiment Analysis

TL;DR

This work tackles multimodal conversational sentiment analysis via two MCABSA challenges: panoptic sentiment sextuple extraction and sentiment flipping analysis. It introduces a structured prompting pipeline with MSGR for robust Target–Aspect extraction, followed by Hybrid LLM Optimization Strategy (HLOS) to refine opinions, sentiments, and rationales, plus an ensemble of three LLMs for flip detection. Empirical results show substantial gains from component-wise refinements (subtask-I) and strong ensemble robustness (subtask-II), achieving 47.38% average and 74.12% Exact Match F1, respectively. The approach demonstrates the feasibility of combining multimodal captioning, stepwise extraction, and model ensembling for rich, multimodal sentiment understanding with potential impact on emotionally intelligent AI systems.

Abstract

Understanding sentiment in multimodal conversations is a complex yet crucial challenge toward building emotionally intelligent AI systems. The Multimodal Conversational Aspect-based Sentiment Analysis (MCABSA) Challenge invited participants to tackle two demanding subtasks: (1) extracting a comprehensive sentiment sextuple, including holder, target, aspect, opinion, sentiment, and rationale from multi-speaker dialogues, and (2) detecting sentiment flipping, which detects dynamic sentiment shifts and their underlying triggers. For Subtask-I, in the present paper, we designed a structured prompting pipeline that guided large language models (LLMs) to sequentially extract sentiment components with refined contextual understanding. For Subtask-II, we further leveraged the complementary strengths of three LLMs through ensembling to robustly identify sentiment transitions and their triggers. Our system achieved a 47.38% average score on Subtask-I and a 74.12% exact match F1 on Subtask-II, showing the effectiveness of step-wise refinement and ensemble strategies in rich, multimodal sentiment analysis tasks.
Paper Structure (11 sections, 3 equations, 5 figures, 2 tables)

This paper contains 11 sections, 3 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Challenge Details
  3. Method
  4. Overvall Architecture
  5. Multimodal Caption Generation
  6. Multi-Sampling Generation and Refinement Method for Target and Aspect Extraction
  7. Hybrid LLM Optimization Strategy for Enhanced Sentiment Sextuple Extraction
  8. Integrated Model Approach for Sentiment Flipping Analysis
  9. Experiments and Results
  10. Discussion
  11. Conclusions

Figures (5)

  • Figure 1: Illustration of the input data format provided in the MCABSA Challenge. The data comprises multi-party, multi-turn dialogues that integrate textual content with multimodal elements, including audio, images, and videos. Speaker information is also included alongside the dialogue content.
  • Figure 2: Overview of the proposed framework for the MCABSA Challenge. The framework consists of three main components: (1) multimodal caption generation, where audio, image, and video descriptions are extracted and integrated into the dialogue input; (2) Panoptic Sentiment Sextuple Extraction, which performs step-wise extraction of sentiment sextuples; and (3) Sentiment Flipping Analysis, which analyzes sentiment flipping through an ensemble approach with three models.
  • Figure 3: Multi-Sampling Generation and Refinement (MSGR) for target-aspect extraction.
  • Figure 4: Hybrid LLM Optimization Strategy (HLOS) for enhanced sentiment sextuple extraction.
  • Figure 5: A representative example illustrating improvements from MSGR and HLOS over the baseline on Subtask-I. Results in red indicate errors made by the baseline (left), while those in green show the corrected outputs produced by our method (right).