Benchmarking Multi-modal Semantic Segmentation under Sensor Failures: Missing and Noisy Modality Robustness

TL;DR

This work tackles the gap between MMSS research and real-world deployment by introducing a dedicated robustness benchmark that evaluates models under three failure modes: Entire-Missing Modality ($EMM$), Random-Missing Modality ($RMM$), and Noisy Modality ($NM$). It develops a probabilistic evaluation framework with two failure assumptions and four metrics, $mIoU^{Avg}_{EMM}$, $mIoU^{E}_{EMM}$, $mIoU^{Avg}_{RMM}$, and $mIoU^{E}_{RMM}$, alongside a robust evaluation protocol on the DELIVER dataset. By benchmarking representative MMSS methods across RGB-centric, equal-contribution, and adaptive-selection fusion strategies, the study reveals that adaptive methods (MAGIC/MAGIC++) achieve superior robustness, while RGB-reliant models suffer under modality loss. The results provide practical guidance for designing resilient MMSS systems and highlight the value of adaptive, cross-modal fusion in real-world sensing environments, with source code available for reproducibility.

Abstract

Multi-modal semantic segmentation (MMSS) addresses the limitations of single-modality data by integrating complementary information across modalities. Despite notable progress, a significant gap persists between research and real-world deployment due to variability and uncertainty in multi-modal data quality. Robustness has thus become essential for practical MMSS applications. However, the absence of standardized benchmarks for evaluating robustness hinders further advancement. To address this, we first survey existing MMSS literature and categorize representative methods to provide a structured overview. We then introduce a robustness benchmark that evaluates MMSS models under three scenarios: Entire-Missing Modality (EMM), Random-Missing Modality (RMM), and Noisy Modality (NM). From a probabilistic standpoint, we model modality failure under two conditions: (1) all damaged combinations are equally probable; (2) each modality fails independently following a Bernoulli distribution. Based on these, we propose four metrics-$mIoU^{Avg}_{EMM}$, $mIoU^{E}_{EMM}$, $mIoU^{Avg}_{RMM}$, and $mIoU^{E}_{RMM}$-to assess model robustness under EMM and RMM. This work provides the first dedicated benchmark for MMSS robustness, offering new insights and tools to advance the field. Source code is available at https://github.com/Chenfei-Liao/Multi-Modal-Semantic-Segmentation-Robustness-Benchmark.

Figures (4)

• Figure 1: History of MMSS methods.
• Figure 2: Framework of our multi-modal semantic segmentation robustness benchmark.
• Figure 3: Visualization of EMM validation results. The scale of the radar chart is set to 5.
• Figure 4: Visualization of RMM validation results with $r=0.25$. The scale of the radar chart is set to 2.