Mix-modal Federated Learning for MRI Image Segmentation

TL;DR

A novel modality decoupling and memorizing mix-modal federated learning framework (MDM-MixMFL) for MRI image segmentation is proposed, characterized by a modality decoupling strategy and a modality memorizing mechanism.

Abstract

Magnetic resonance imaging (MRI) image segmentation is crucial in diagnosing and treating many diseases, such as brain tumors. Existing MRI image segmentation methods mainly fall into a centralized multimodal paradigm, which is inapplicable in engineering non-centralized mix-modal medical scenarios. In this situation, each distributed client (hospital) processes multiple mixed MRI modalities, and the modality set and image data for each client are diverse, suffering from extensive client-wise modality heterogeneity and data heterogeneity. In this paper, we first formulate non-centralized mix-modal MRI image segmentation as a new paradigm for federated learning (FL) that involves multiple modalities, called mix-modal federated learning (MixMFL). It distinguishes from existing multimodal federating learning (MulMFL) and cross-modal federating learning (CroMFL) paradigms. Then, we proposed a novel modality decoupling and memorizing mix-modal federated learning framework (MDM-MixMFL) for MRI image segmentation, which is characterized by a modality decoupling strategy and a modality memorizing mechanism. Specifically, the modality decoupling strategy disentangles each modality into modality-tailored and modality-shared information. During mix-modal federated updating, corresponding modality encoders undergo tailored and shared updating, respectively. It facilitates stable and adaptive federating aggregation of heterogeneous data and modalities from distributed clients. Besides, the modality memorizing mechanism stores client-shared modality prototypes dynamically refreshed from every modality-tailored encoder to compensate for incomplete modalities in each local client.

Figures (7)

• Figure 1: Paradigm comparison of federated learning involving multiple modalities. FLAIR, T1c, T1, and T2 denote different modalities of MRI images, respectively. (a) MulMFL: All multimodal data in the same image modalities but from different data distributions (different hospitals), containing data heterogeneity. (b) CroMFL: Each client holds one different modality from the same data distribution, containing modality heterogeneity. (c) MixMFL: Each client holds multiple mixed modalities and also from different data distributions, containing both modality heterogeneity and data heterogeneity.
• Figure 2: Overview pipeline of the proposed MDM-MixMFL framework. (a) The federated parameter transmission and local feature transform processes among three distributed clients (i, j, k). (b) The detailed network structure for every client.
• Figure 3: The proposed modality decoupler consists of two main branches accompanied by two different losses. The above branch performs the modality classification guided by a cross-entropy loss $\mathcal{L}_{cls}$. The bottom branch performs grouping and merging of modality representations guided by a triplet loss $\mathcal{L}_{tri}$. GAP denotes the global average pooling layer. GRL denotes the gradient reversal layer, which reverses the parameter gradients and forces the layers below the GRL layer to update along the opposite direction.
• Figure 4: Venn diagram for relation illustration of multiple representation distributions. The symbols $P$, $Q$, and $R$ denote three representation modalities from one local client, respectively.
• Figure 5: Illustration of the memorizing, refreshing, and retrieving processes of the proposed modality memory mechanism.