Intelligent Communication Mixture-of-Experts Boosted-Medical Image Segmentation Foundation Model
Xinwei Zhang, Hu Chen, Zhe Yuan, Sukun Tian, Peng Feng
TL;DR
The work tackles two key drawbacks of parameter-efficient fine-tuning for medical image segmentation foundation models: insufficient high-level feature representation and disruption of pretrained weights. It introduces IC-MoE, a three-expert mixture with a pixel-probability adaptive voting fusion and semantic-guided contrastive learning to boost high-level semantics while preserving pretrained integrity. Across ISIC, BUSI, and GLAS, IC-MoE demonstrates state-of-the-art segmentation performance and strong generalizability, validating the proposed fusion and contrastive strategies. The approach offers a practical path to enhancing clinical segmentation tasks without sacrificing the benefits of pretrained weight structures, albeit with higher training cost and complexity.
Abstract
Foundation models for medical image segmentation have achieved remarkable performance. Adaptive fine-tuning of natural image segmentation foundation models is crucial for medical image segmentation tasks. However, some limitations exist in existing fine-tuning methods: 1) insufficient representation of high-level features and 2) the fine-tuning process disrupts the structural integrity of pretrained weights. Inspired by these critical problems, we propose an intelligent communication mixture-of-experts boosted-medical image segmentation foundation model, named IC-MoE, with twofold ideas: 1) We construct basic experts, semantic experts, and adaptive experts. Moreover, we implement a pixel probability adaptive voting strategy, which enables expert selection and fusion through label consistency and load balancing. This approach preliminarily enhances the representation capability of high-level features while preserving the structural integrity of pretrained weights. 2) We propose a semantic-guided contrastive learning method to address the issue of weak supervision in contrastive learning. This method further enhances the representation capability of high-level features while preserving the structural integrity of pretrained weights. Extensive experiments across three public medical image segmentation datasets demonstrate that the IC-MoE outperforms other SOTA models. Consequently, the proposed IC-MoE effectively supplements foundational medical image segmentation models with high-level features and pretrained structural integrity. We also validate the superior generalizability of the IC-MoE across diverse medical image segmentation scenarios.