MissBench: Benchmarking Multimodal Affective Analysis under Imbalanced Missing Modalities

TL;DR

Experiments on representative method families show that models that appear robust under shared missing rates can still exhibit marked modality inequity and optimization imbalance under imbalanced conditions, positioning MissBench, together with MEI and MLI, as practical tools for stress-testing and analyzing multimodal affective models in realistic incomplete-modality settings.

Abstract

Multimodal affective computing underpins key tasks such as sentiment analysis and emotion recognition. Standard evaluations, however, often assume that textual, acoustic, and visual modalities are equally available. In real applications, some modalities are systematically more fragile or expensive, creating imbalanced missing rates and training biases that task-level metrics alone do not reveal. We introduce MissBench, a benchmark and framework for multimodal affective tasks that standardizes both shared and imbalanced missing-rate protocols on four widely used sentiment and emotion datasets. MissBench also defines two diagnostic metrics. The Modality Equity Index (MEI) measures how fairly different modalities contribute across missing-modality configurations. The Modality Learning Index (MLI) quantifies optimization imbalance by comparing modality-specific gradient norms during training, aggregated across modality-related modules. Experiments on representative method families show that models that appear robust under shared missing rates can still exhibit marked modality inequity and optimization imbalance under imbalanced conditions. These findings position MissBench, together with MEI and MLI, as practical tools for stress-testing and analyzing multimodal affective models in realistic incomplete-modality settings.For reproducibility, we release our code at: https://anonymous.4open.science/r/MissBench-4098/

Figures (14)

• Figure 1: Sampled modality-specific missing rate under mean-match SMR=$0.5$ and IMR=$(0.4,0.5,0.6)$. Under SMR $\hat{r_m}\approx r_{\mathrm{sh}}$, while under IMR $r_m$ can receive any predefined value $\in[0,1)$.
• Figure 2: Decoupled modality parameters wang2023unlocking of RedCore sun2024redcore on IEMOCAP under different missing-rate configurations, illustrating how a single dominant modality can drive most parameter updates under imbalanced missing rates.
• Figure 3: MissBench benchmark pipeline for multimodal affective tasks, illustrating data preparation, SMR/IMR missingness protocols, unified training with model plugins, and evaluation with both task metrics and modality-aware diagnostics.
• Figure 4: Performance of multimodal methods under shared missing rates (SMR) on IEMOCAP (WA) and CMU-MOSI/MOSEI/CH-SIMS (Acc-2). IMR-aware methods are shown in red, missing-modality handling models in blue, and gradient-based or generic baselines in green.
• Figure 5: Statistics of GCNet lian2023gcnet during training on IEMOCAP under different missing configurations.