DC-Seg: Disentangled Contrastive Learning for Brain Tumor Segmentation with Missing Modalities

TL;DR

Missing-modality brain MRI segmentation is a practical challenge. DC-Seg tackles this with a disentangled framework that performs bidirectional contrastive learning to obtain modality-invariant anatomical representations and modality-specific features, guided by a reconstruction path and a segmentation-based regularizer. Key contributions include anatomical and modality contrastive losses at the 3D image level, a regularizer that enforces multi-modality usage, and strong results on BraTS 2020 with good generalization to WMH. This approach yields robust segmentation under missing modalities and provides code for reproducibility at the provided GitHub URL.

Abstract

Accurate segmentation of brain images typically requires the integration of complementary information from multiple image modalities. However, clinical data for all modalities may not be available for every patient, creating a significant challenge. To address this, previous studies encode multiple modalities into a shared latent space. While somewhat effective, it remains suboptimal, as each modality contains distinct and valuable information. In this study, we propose DC-Seg (Disentangled Contrastive Learning for Segmentation), a new method that explicitly disentangles images into modality-invariant anatomical representation and modality-specific representation, by using anatomical contrastive learning and modality contrastive learning respectively. This solution improves the separation of anatomical and modality-specific features by considering the modality gaps, leading to more robust representations. Furthermore, we introduce a segmentation-based regularizer that enhances the model's robustness to missing modalities. Extensive experiments on the BraTS 2020 and a private white matter hyperintensity(WMH) segmentation dataset demonstrate that DC-Seg outperforms state-of-the-art methods in handling incomplete multimodal brain tumor segmentation tasks with varying missing modalities, while also demonstrate strong generalizability in WMH segmentation. The code is available at https://github.com/CuCl-2/DC-Seg.

Figures (3)

• Figure 1: Overview of DC-Seg, which disentangles images from different modalities into anatomical and modality representations using bidirectional contrastive learning, and fuses modality-invariant anatomical representations for the downstream tumor segmentation task. For clarity in the figure, $D^{\text{sep}}$ is omitted.
• Figure 2: Visualization of the input modalities, our predicted segmentation maps, and MedSAM prediction with bounding box prompt.
• Figure 3: Visualization of anatomical and modality representations on the unseen test set of BRATS.