MissMAC-Bench: Building Solid Benchmark for Missing Modality Issue in Robust Multimodal Affective Computing

TL;DR

MissMAC-Bench addresses the missing modality issue in multimodal affective computing by proposing a unified benchmark with two core principles: no missing priors during training and a single model capable of handling both complete and incomplete inputs. It introduces fixed and random missing protocols at dataset and instance levels to reflect real-world conditions and evaluates across two subtasks (MSA and MER) on four datasets using three language models. The results show that adaptation-based and generation-based methods offer the strongest robustness under diverse missing patterns, while alignment-based and heavy data augmentation approaches can be less reliable when modalities are incomplete or imbalanced. The benchmark provides a practical, cross-dataset framework to quantify inter-modal robustness and cross-modal synergy, with a strong emphasis on fair comparison and generalization, supporting the development of more resilient MAC systems in real-world multimedia data mining.

Abstract

As a knowledge discovery task over heterogeneous data sources, current Multimodal Affective Computing (MAC) heavily rely on the completeness of multiple modalities to accurately understand human's affective state. However, in real-world scenarios, the availability of modality data is often dynamic and uncertain, leading to substantial performance fluctuations due to the distribution shifts and semantic deficiencies of the incomplete multimodal inputs. Known as the missing modality issue, this challenge poses a critical barrier to the robustness and practical deployment of MAC models. To systematically quantify this issue, we introduce MissMAC-Bench, a comprehensive benchmark designed to establish fair and unified evaluation standards from the perspective of cross-modal synergy. Two guiding principles are proposed, including no missing prior during training, and one single model capable of handling both complete and incomplete modality scenarios, thereby ensuring better generalization. Moreover, to bridge the gap between academic research and real-world applications, our benchmark integrates evaluation protocols with both fixed and random missing patterns at the dataset and instance levels. Extensive experiments conducted on 3 widely-used language models across 4 datasets validate the effectiveness of diverse MAC approaches in tackling the missing modality issue. Our benchmark provides a solid foundation for advancing robust multimodal affective computing and promotes the development of multimedia data mining.

Figures (6)

• Figure 1: Missing modality issue for robust multimodal affective computing in downstream application.
• Figure 2: Taxonomy for incomplete multimodal input with diverse missing degrees and task generalization ability.
• Figure 3: Illustration of random missing protocol, including dataset-level and instance-level evaluation.
• Figure 4: Illustration of $C\&R$ dimensions for one model.
• Figure 5: Evaluation on $C\&R$ Dimension with BERT/sBERT on Multimodal Sentiment Analysis (MSA) and Multimodal Emotion Recognition (MER) subtasks at both dataset-level ($MR$) and instance-level ($MP$) random missing protocols.