S&D Messenger: Exchanging Semantic and Domain Knowledge for Generic Semi-Supervised Medical Image Segmentation

TL;DR

This paper tackles the challenge of unifying three related medical image segmentation tasks—SSMIS, UMDA, and Semi-MDG—under a single semi-supervised framework. It introduces the Semantic & Domain Knowledge Messenger (S&D Messenger), which enables bidirectional knowledge transfer between the labeled (semantic knowledge) and unlabeled (domain knowledge) data flows via L2U patch-based knowledge delivery and U2L cross-attention-based Messenger Transformer blocks. By incorporating these knowledge deliveries into a naive pseudo-labeling baseline, the approach achieves state-of-the-art performance across six datasets and all three task categories, notably delivering substantial Dice gains at low labeling ratios. The method promises broad practical impact for robust, label-efficient training of Transformer-based models in multi-domain medical imaging contexts and beyond.

Abstract

Semi-supervised medical image segmentation (SSMIS) has emerged as a promising solution to tackle the challenges of time-consuming manual labeling in the medical field. However, in practical scenarios, there are often domain variations within the datasets, leading to derivative scenarios like semi-supervised medical domain generalization (Semi-MDG) and unsupervised medical domain adaptation (UMDA). In this paper, we aim to develop a generic framework that masters all three tasks. We notice a critical shared challenge across three scenarios: the explicit semantic knowledge for segmentation performance and rich domain knowledge for generalizability exclusively exist in the labeled set and unlabeled set respectively. Such discrepancy hinders existing methods from effectively comprehending both types of knowledge under semi-supervised settings. To tackle this challenge, we develop a Semantic & Domain Knowledge Messenger (S&D Messenger) which facilitates direct knowledge delivery between the labeled and unlabeled set, and thus allowing the model to comprehend both of them in each individual learning flow. Equipped with our S&D Messenger, a naive pseudo-labeling method can achieve huge improvement on six benchmark datasets for SSMIS (+7.5%), UMDA (+5.6%), and Semi-MDG tasks (+1.14%), compared with state-of-the-art methods designed for specific tasks.

Figures (8)

• Figure 1: (a): Compared with SOTA methods tailored for the specific scenario, GenericSSL wang2024towards achieved great improvement on SSMIS (+11.85% Dice), however the results on UMDA (+4.9% Dice) and Semi-MDG (-2.15% Dice) can only match or even lag behind. (b): Performance of typical SSMIS methods (UA-MT yu2019uamt, CLD lin2022cld, and Pseudo-labeling lee2013pseudo) on UMDA task (MMWHS dataset) with different backbones. These methods, with or without Transformer-based backbone, i.e., SegFormer xie2021segformer, encounter severe performance drops on the UMDA task, showing a large margin compared with the task-tailored SOTA.
• Figure 2: The small labeled sets have explicit semantic knowledge derived from class-wise labels but lack comprehensive domain knowledge, i.e., feature variations across different domains, due to insufficient or entirely absent data from the target domain. Conversely, the large unlabeled sets exhibit the opposite characteristics, with abundant domain knowledge but lacking explicit semantic information.
• Figure 3: Insight of S&D-Messenger: enabling knowledge delivery across the learning flow of the labeled and unlabeled set.
• Figure 4: Illustration of our proposed S&D-Messenger for SSMIS. Our S&D-Messenger inherits the architecture from pseudo-labeling with SegFormer as backbone xie2021segformer, while we replace the original Transformer Block with our S&D-Messenger which employs a novel feature-intervention attention module to use diverse unlabeled data to intervene in the learning flow of the labeled data.
• Figure 5: Qualitative comparison between S&D-Messenger and the SOTA methods on 20% labeled Synapse dataset.