Double Banking on Knowledge: Customized Modulation and Prototypes for Multi-Modality Semi-supervised Medical Image Segmentation

TL;DR

A modality all-in-one segmentation network that accommodates data from any number of modalities, removing the limitation on modality count is proposed and a Dual Consistency (DC) strategy is introduced that enforces consistency at both the image and feature levels without relying on generative methods.

Abstract

Multi-modality (MM) semi-supervised learning (SSL) based medical image segmentation has recently gained increasing attention for its ability to utilize MM data and reduce reliance on labeled images. However, current methods face several challenges: (1) Complex network designs hinder scalability to scenarios with more than two modalities. (2) Focusing solely on modality-invariant representation while neglecting modality-specific features, leads to incomplete MM learning. (3) Leveraging unlabeled data with generative methods can be unreliable for SSL. To address these problems, we propose Double Bank Dual Consistency (DBDC), a novel MM-SSL approach for medical image segmentation. To address challenge (1), we propose a modality all-in-one segmentation network that accommodates data from any number of modalities, removing the limitation on modality count. To address challenge (2), we design two learnable plug-in banks, Modality-Level Modulation bank (MLMB) and Modality-Level Prototype (MLPB) bank, to capture both modality-invariant and modality-specific knowledge. These banks are updated using our proposed Modality Prototype Contrastive Learning (MPCL). Additionally, we design Modality Adaptive Weighting (MAW) to dynamically adjust learning weights for each modality, ensuring balanced MM learning as different modalities learn at different rates. Finally, to address challenge (3), we introduce a Dual Consistency (DC) strategy that enforces consistency at both the image and feature levels without relying on generative methods. We evaluate our method on a 2-to-4 modality segmentation task using three open-source datasets, and extensive experiments show that our method outperforms state-of-the-art approaches.

Figures (8)

• Figure 1: (a) Transfer learning task. (b) Single modality SSL task. (c) MM-SSL task.
• Figure 2: The structure of our proposed modality-all-in-one network, which is built on a vanilla U-Net, plug-in with the Modality-Level Modulation Bank (MLMB) and Modality-Level Prototype Bank (MLPB).
• Figure 3: Pipeline of our proposed Double Bank Dual Consistency (DBDC) framework.
• Figure 4: 2D and 3D segmentation results on the 2-modality dataset.
• Figure 5: Segmentation results on the abdominal dataset.