Reducing Unimodal Bias in Multi-Modal Semantic Segmentation with Multi-Scale Functional Entropy Regularization

TL;DR

A simple but effective plug-and-play regularization term based on functional entropy is applied, designed to intuitively balance the contribution of each visual modality to the segmentation results, which mitigates unimodal dominance and establishes a more balanced and robust segmentation framework.

Abstract

Fusing and balancing multi-modal inputs from novel sensors for dense prediction tasks, particularly semantic segmentation, is critically important yet remains a significant challenge. One major limitation is the tendency of multi-modal frameworks to over-rely on easily learnable modalities, a phenomenon referred to as unimodal dominance or bias. This issue becomes especially problematic in real-world scenarios where the dominant modality may be unavailable, resulting in severe performance degradation. To this end, we apply a simple but effective plug-and-play regularization term based on functional entropy, which introduces no additional parameters or modules. This term is designed to intuitively balance the contribution of each visual modality to the segmentation results. Specifically, we leverage the log-Sobolev inequality to bound functional entropy using functional-Fisher-information. By maximizing the information contributed by each visual modality, our approach mitigates unimodal dominance and establishes a more balanced and robust segmentation framework. A multi-scale regularization module is proposed to apply our proposed plug-and-play term on high-level features and also segmentation predictions for more balanced multi-modal learning. Extensive experiments on three datasets demonstrate that our proposed method achieves superior performance, i.e., +13.94%, +3.25%, and +3.64%, without introducing any additional parameters.

Figures (7)

• Figure 1: (a) Previous methods use complex architectures to fuse multi-modal data. (b) Our approach simplifies the architecture and use plug-and-play regularization terms. The results show significant improvements of ours than CMNeXt zhang2023delivering and MAGIC zheng2025centering.
• Figure 2: Visualization of Fisher information during training with real-world MUSES (right) and synthetic DELIVER (left) datasets. Adding a regularization term reduces the gap between modality Fisher information values. With regularization, the model utilizes both input modalities, resulting in closer modality distances. Without regularization, the model relies more on RGB in the first two rows and Event in the last two, increasing the modality distance (red arrows).
• Figure 3: (a) The proposed regularization term for multi-modal input and (2) overall framework of our proposed multi-scale regularization terms' implementation.
• Figure 4: Visualization on DELIVER zhang2023delivering with zhang2023delivering and zheng2025centering.
• Figure 5: Comparison of model performance with (w/) and without (w/o) our proposed prediction level regularization term across different modalities (RGB-Depth, RGB, and RGB-Event).