Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillation

TL;DR

This work tackles unimodal bias in multimodal semantic segmentation by introducing AnySeg, a framework that learns robust anymodal segmentors via a strong multimodal teacher trained with Parallel Multimodal Learning (PML) and a student guided through Unimodal Distillation (UMD), Cross-modal Distillation (CMD), and modality-agnostic semantic distillation (MAD). The method transfers knowledge across multi-scale features and prediction maps, while simulated missing modalities through anymodal dropout to mirror real-world sensor absence. Empirical results on synthetic and real benchmarks (MUSES and DELIVER) show state-of-the-art improvements in mean IoU and balanced performance across modalities, with notable gains in challenging inputs like Event and LiDAR. Overall, the approach demonstrates that modality-aware distillation and prediction-level transfer can substantially mitigate unimodal bias and enable robust, modality-agnostic segmentation in diverse sensing scenarios, albeit with additional training cost.

Abstract

Simultaneously using multimodal inputs from multiple sensors to train segmentors is intuitively advantageous but practically challenging. A key challenge is unimodal bias, where multimodal segmentors over rely on certain modalities, causing performance drops when others are missing, common in real world applications. To this end, we develop the first framework for learning robust segmentor that can handle any combinations of visual modalities. Specifically, we first introduce a parallel multimodal learning strategy for learning a strong teacher. The cross-modal and unimodal distillation is then achieved in the multi scale representation space by transferring the feature level knowledge from multimodal to anymodal segmentors, aiming at addressing the unimodal bias and avoiding over-reliance on specific modalities. Moreover, a prediction level modality agnostic semantic distillation is proposed to achieve semantic knowledge transferring for segmentation. Extensive experiments on both synthetic and real-world multi-sensor benchmarks demonstrate that our method achieves superior performance.

Figures (6)

• Figure 1: (a) Overall of AnySeg with a two-stage training strategy: the multimodal teacher is first trained using PML, then frozen for student distillation. (b) Unimodal feature distillation transfers intra-modality knowledge. (c) Cross-modal feature distillation enables modality interaction transfer.
• Figure 2: PML for learning a strong multimodal segmentor as teacher model.
• Figure 3: Qualitative comparison on MUSES.
• Figure 4: TSNE visualization of multi-modal features (RGB-R, Depth-D, Event-E, and LiDAR-L) and the learned features of our AnySeg framework.
• Figure 5: Additional qualitative comparisons that highlight our method’s robustness under diverse dropout conditions.