Federated Client-tailored Adapter for Medical Image Segmentation

TL;DR

This work tackles the challenge of privacy-preserving medical image segmentation across heterogeneous data islands by introducing a Federated Client-tailored Adapter (FCA). FCA leverages off-the-shelf medical foundation models with lightweight adapters and uses a Global-local Decomposer to separate common and individual parameter components, enabling stable, client-specific segmentation models. It introduces two updating strategies, Binary Federated Updating (BFU) and Smooth Federated Updating (SFU), to realize fine-grained, client-tailored federated learning and mitigate client drift. Extensive experiments on CXRS-HG, HLS, and AMD-SD-HG show that FCA achieves state-of-the-art performance with robust handling of class imbalance and distribution diversity, while generalizing across multiple MFMs and maintaining practical efficiency.

Abstract

Medical image segmentation in X-ray images is beneficial for computer-aided diagnosis and lesion localization. Existing methods mainly fall into a centralized learning paradigm, which is inapplicable in the practical medical scenario that only has access to distributed data islands. Federated Learning has the potential to offer a distributed solution but struggles with heavy training instability due to client-wise domain heterogeneity (including distribution diversity and class imbalance). In this paper, we propose a novel Federated Client-tailored Adapter (FCA) framework for medical image segmentation, which achieves stable and client-tailored adaptive segmentation without sharing sensitive local data. Specifically, the federated adapter stirs universal knowledge in off-the-shelf medical foundation models to stabilize the federated training process. In addition, we develop two client-tailored federated updating strategies that adaptively decompose the adapter into common and individual components, then globally and independently update the parameter groups associated with common client-invariant and individual client-specific units, respectively. They further stabilize the heterogeneous federated learning process and realize optimal client-tailored instead of sub-optimal global-compromised segmentation models. Extensive experiments on three large-scale datasets demonstrate the effectiveness and superiority of the proposed FCA framework for federated medical segmentation.

Figures (6)

• Figure 1: Conventional learning paradigms for tackling heterogeneous distributed medical data. (a) Centralized learning aggregates all data together to train a single model. (b) Conventional federated learning trains a global-compromised model for all clients without sharing sensitive local data. (c) The proposed client-tailored federated learning trains client-customized models for each client without sharing sensitive local data.
• Figure 2: (a) Client-wise heterogeneity in X-ray chest images consists of common class imbalance (long-tail distribution) and various distribution diversity. (b) In heterogeneous distributed scenarios, conventional federated learning (e.g. FedAvg*r1) suffers from considerable instability and slow convergence while our federated client-tailored adapter (FCA-SFU) effectively alleviates this issue. The transparent lines represent the original experimental results, while the solid lines represent smoothed results that facilitate visualization.
• Figure 3: (a) Overview of the proposed federated client-tailored adapter (FCA) framework. (b) Detailed structure of the client-adaptive adapter. (c) The mechanism of the Global-local decomposer (GLD).
• Figure 4: (a) The binary federated updating (BFU) strategy binary distinguishes the adapter units in each client into local client-specific and global client-invariant units and respectively updates them locally and globally. (b) The smooth federated updating (SFU) strategy probabilistically distributes each unit to all clients and thus each unit probabilistically participates in the federated parameter updating of all clients.
• Figure 5: Visualization comparison of heterogeneous federated segmentation results on the CXRS-HG dataset