Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DaQ-MSA: Denoising and Qualifying Diffusion Augmentations for Multimodal Sentiment Analysis

Jiazhang Liang, Jianheng Dai, Miaosen Luo, Menghua Jiang, Sijie Mai

TL;DR

The paper tackles data scarcity in multimodal sentiment analysis by integrating diffusion-based Video and Audio augmentation with a quality-aware weighting strategy. A generate-evaluate-reweight pipeline produces semantically-preserving augmentations, which are scored by a decoupled QA module and weighted during fine-tuning of a multimodal large language model, enabling annotation-free, data-efficient learning. Empirical results across CH-SIMS, CMU-MOSI, and MUStARD show state-of-the-art performance and robustness, with ablations confirming the value of QA-based filtering and diffusion augmentation. This approach offers a scalable pathway to leverage synthetic multimodal data for reliable sentiment understanding in real-world, low-resource settings.

Abstract

Multimodal large language models (MLLMs) have demonstrated strong performance on vision-language tasks, yet their effectiveness on multimodal sentiment analysis remains constrained by the scarcity of high-quality training data, which limits accurate multimodal understanding and generalization. To alleviate this bottleneck, we leverage diffusion models to perform semantics-preserving augmentation on the video and audio modalities, expanding the multimodal training distribution. However, increasing data quantity alone is insufficient, as diffusion-generated samples exhibit substantial quality variation and noisy augmentations may degrade performance. We therefore propose DaQ-MSA (Denoising and Qualifying Diffusion Augmentations for Multimodal Sentiment Analysis), which introduces a quality scoring module to evaluate the reliability of augmented samples and assign adaptive training weights. By down-weighting low-quality samples and emphasizing high-fidelity ones, DaQ-MSA enables more stable learning. By integrating the generative capability of diffusion models with the semantic understanding of MLLMs, our approach provides a robust and generalizable automated augmentation strategy for training MLLMs without any human annotation or additional supervision.

DaQ-MSA: Denoising and Qualifying Diffusion Augmentations for Multimodal Sentiment Analysis

TL;DR

The paper tackles data scarcity in multimodal sentiment analysis by integrating diffusion-based Video and Audio augmentation with a quality-aware weighting strategy. A generate-evaluate-reweight pipeline produces semantically-preserving augmentations, which are scored by a decoupled QA module and weighted during fine-tuning of a multimodal large language model, enabling annotation-free, data-efficient learning. Empirical results across CH-SIMS, CMU-MOSI, and MUStARD show state-of-the-art performance and robustness, with ablations confirming the value of QA-based filtering and diffusion augmentation. This approach offers a scalable pathway to leverage synthetic multimodal data for reliable sentiment understanding in real-world, low-resource settings.

Abstract

Multimodal large language models (MLLMs) have demonstrated strong performance on vision-language tasks, yet their effectiveness on multimodal sentiment analysis remains constrained by the scarcity of high-quality training data, which limits accurate multimodal understanding and generalization. To alleviate this bottleneck, we leverage diffusion models to perform semantics-preserving augmentation on the video and audio modalities, expanding the multimodal training distribution. However, increasing data quantity alone is insufficient, as diffusion-generated samples exhibit substantial quality variation and noisy augmentations may degrade performance. We therefore propose DaQ-MSA (Denoising and Qualifying Diffusion Augmentations for Multimodal Sentiment Analysis), which introduces a quality scoring module to evaluate the reliability of augmented samples and assign adaptive training weights. By down-weighting low-quality samples and emphasizing high-fidelity ones, DaQ-MSA enables more stable learning. By integrating the generative capability of diffusion models with the semantic understanding of MLLMs, our approach provides a robust and generalizable automated augmentation strategy for training MLLMs without any human annotation or additional supervision.
Paper Structure (20 sections, 12 equations, 5 figures, 5 tables)

This paper contains 20 sections, 12 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Sentiment Analysis
  4. Data Augmentation and Diffusion Models
  5. Method
  6. Overall Framework
  7. Diffusion-Based Data Generation
  8. Quality-Aware Scoring Mechanism
  9. Weighted Fine-tuning
  10. Experiments
  11. Comparison with the State-of-the-Art
  12. Ablation Study
  13. Interpretability of QA Module.
  14. Representation Learning Visualization
  15. Conclusion
  16. ...and 5 more sections

Figures (5)

  • Figure 1: Overview of the proposed DaQ-MSA framework.
  • Figure 2: Architecture of the quality-aware training pipeline.
  • Figure 3: Qualitative visualization of the QA module's scoring mechanism.
  • Figure 4: t-SNE visualization of sentiment representations under different training configurations.
  • Figure 5: QA score distributions and adaptive weight mapping. (a) Score distributions for original and augmented samples. (b) Mapping from QA scores to training weights (Pearson $r=0.91$).