PASSION: Towards Effective Incomplete Multi-Modal Medical Image Segmentation with Imbalanced Missing Rates

TL;DR

PASSION tackles incomplete multi-modal medical image segmentation with imbalanced missing rates by introducing pixel-wise and semantic-wise self-distillation to align uni-modal optimization with multi-modal knowledge, and a relative-preference regularization to balance learning paces across modalities. The method yields a plug-and-play solution that improves performance across backbones on BraTS2020 and MyoPS2020, outperforming existing modality-balancing approaches. By formalizing IDT and leveraging both per-pixel and class-prototype transfers, PASSION achieves robust, balanced segmentation in realistic training scenarios while maintaining practical applicability. Code and experiments demonstrate broad utility for clinical deployment where modality availability varies.

Abstract

Incomplete multi-modal image segmentation is a fundamental task in medical imaging to refine deployment efficiency when only partial modalities are available. However, the common practice that complete-modality data is visible during model training is far from realistic, as modalities can have imbalanced missing rates in clinical scenarios. In this paper, we, for the first time, formulate such a challenging setting and propose Preference-Aware Self-diStillatION (PASSION) for incomplete multi-modal medical image segmentation under imbalanced missing rates. Specifically, we first construct pixel-wise and semantic-wise self-distillation to balance the optimization objective of each modality. Then, we define relative preference to evaluate the dominance of each modality during training, based on which to design task-wise and gradient-wise regularization to balance the convergence rates of different modalities. Experimental results on two publicly available multi-modal datasets demonstrate the superiority of PASSION against existing approaches for modality balancing. More importantly, PASSION is validated to work as a plug-and-play module for consistent performance improvement across different backbones. Code is available at https://github.com/Jun-Jie-Shi/PASSION.

Figures (6)

• Figure 1: Comparison of modality settings in incomplete multi-modal medical image segmentation. $PDT$ denotes perfect data training where modalities share equal missing rates and masked modalities can be visible during training. $IDT$ denotes imperfect data training, formulated in this paper, where modalities own imbalanced missing rates and missing modalities are invisible during training. Incomplete multi-modal sets $D^1, D^2, D^3$, and $D^4$ are generated/simulated from full-modality data $D$ in $PDT$ and drawn from incomplete-modality data $D$ following imbalanced missing rates in $IDT$.
• Figure 2: Illustration of PASSION. $\mathcal{L}^m_{proto}$ and $\mathcal{L}^m_{pixel}$ represent multi-uni self-distillation and $\delta^m$ and $\beta^m$ represent preference-aware regularization. $\blacktriangle$, $\bigstar$, and $\blacksquare$ represent prototypes in Eq. \ref{['eq:proto']} while $Sim$ denotes feature-prototype similarity in Eq. \ref{['eq:sim']}
• Figure 3: Visualized average performance comparison on BraTS2020 and MyoPS2020 given ($s=0.2$, $m=0.5$, $l=0.8$) and ($s=0.3$, $m=0.5$, $l=0.7$) respectively.
• Figure 4: Exemplar segmentation results on BraTS2020 under four modality combinations given $MR=(0.2, 0.4, 0.6, 0.8)$ for T1, T1c, Flair, and T2 respectively.
• Figure 5: Exemplar segmentation results on MyoPS2020 under three modality combinations given $MR=(0.3, 0.5, 0.7)$ for bSSFP, LGE, and T2 respectively.