DSFedMed: Dual-Scale Federated Medical Image Segmentation via Mutual Distillation Between Foundation and Lightweight Models

TL;DR

DSFedMed tackles the challenge of deploying large foundation models for medical image segmentation in privacy-preserving, resource-constrained federated settings. It introduces a dual-scale framework that couples a server-side foundation model with lightweight client models through mutual distillation, aided by controllable synthetic data generated via ControlNet and a learnability-guided sample selection strategy. The method enables efficient, asynchronous federated training, achieving strong segmentation performance with significantly reduced communication and inference costs compared to foundation-model baselines. The results on five non-IID medical datasets show DSFedMed surpasses lightweight baselines and closes the gap to foundation models, offering a scalable path for privacy-aware deployment of large models in healthcare, with potential extensions to multimodal data.

Abstract

Foundation Models (FMs) have demonstrated strong generalization across diverse vision tasks. However, their deployment in federated settings is hindered by high computational demands, substantial communication overhead, and significant inference costs. We propose DSFedMed, a dual-scale federated framework that enables mutual knowledge distillation between a centralized foundation model and lightweight client models for medical image segmentation. To support knowledge distillation, a set of high-quality medical images is generated to replace real public datasets, and a learnability-guided sample selection strategy is proposed to enhance efficiency and effectiveness in dual-scale distillation. This mutual distillation enables the foundation model to transfer general knowledge to lightweight clients, while also incorporating client-specific insights to refine the foundation model. Evaluations on five medical imaging segmentation datasets show that DSFedMed achieves an average 2 percent improvement in Dice score while reducing communication costs and inference time by nearly 90 percent compared to existing federated foundation model baselines. These results demonstrate significant efficiency gains and scalability for resource-limited federated deployments.

Figures (8)

• Figure 1: Comparison of federated segmentation methods in terms of communication cost and Dice performance. Bubble size represents inference time. Foundation model-based methods (e.g., FedSAM) achieve high accuracy but suffer from heavy communication and inference overhead. Lightweight models (e.g., FedU-Net) are efficient but sacrifice accuracy. Our proposed method, DSFedMed, achieves a better balance, offering both high accuracy and low computational and communication costs.
• Figure 2: The Overview of DSFedMed.
• Figure 3: Illustration of proposed learnability-guided mutual distillation.
• Figure 4: Examples of Learnability-Guided Selection.
• Figure 5: The example of datasets and the comparison in the results of SAM, FedSAM, FedMSA, FedU-Net, FednnU-Net, FedTinySAM and DSFedMed. The average Dice of the method on the dataset is marked in the bottom right corner.