Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Modality-specific Encoders and Multimodal Anchors for Personalized Brain Tumor Segmentation

Qian Dai, Dong Wei, Hong Liu, Jinghan Sun, Liansheng Wang, Yefeng Zheng

TL;DR

This work proposes a new FL framework with federated modality-specific encoders and multimodal anchors (FedMEMA) to simultaneously address the two concurrent issues of inter-modal heterogeneity and outperforms various up-to-date methods for multimodal and personalized FL.

Abstract

Most existing federated learning (FL) methods for medical image analysis only considered intramodal heterogeneity, limiting their applicability to multimodal imaging applications. In practice, it is not uncommon that some FL participants only possess a subset of the complete imaging modalities, posing inter-modal heterogeneity as a challenge to effectively training a global model on all participants' data. In addition, each participant would expect to obtain a personalized model tailored for its local data characteristics from the FL in such a scenario. In this work, we propose a new FL framework with federated modality-specific encoders and multimodal anchors (FedMEMA) to simultaneously address the two concurrent issues. Above all, FedMEMA employs an exclusive encoder for each modality to account for the inter-modal heterogeneity in the first place. In the meantime, while the encoders are shared by the participants, the decoders are personalized to meet individual needs. Specifically, a server with full-modal data employs a fusion decoder to aggregate and fuse representations from all modality-specific encoders, thus bridging the modalities to optimize the encoders via backpropagation reversely. Meanwhile, multiple anchors are extracted from the fused multimodal representations and distributed to the clients in addition to the encoder parameters. On the other end, the clients with incomplete modalities calibrate their missing-modal representations toward the global full-modal anchors via scaled dot-product cross-attention, making up the information loss due to absent modalities while adapting the representations of present ones. FedMEMA is validated on the BraTS 2020 benchmark for multimodal brain tumor segmentation. Results show that it outperforms various up-to-date methods for multimodal and personalized FL and that its novel designs are effective. Our code is available.

Federated Modality-specific Encoders and Multimodal Anchors for Personalized Brain Tumor Segmentation

TL;DR

This work proposes a new FL framework with federated modality-specific encoders and multimodal anchors (FedMEMA) to simultaneously address the two concurrent issues of inter-modal heterogeneity and outperforms various up-to-date methods for multimodal and personalized FL.

Abstract

Most existing federated learning (FL) methods for medical image analysis only considered intramodal heterogeneity, limiting their applicability to multimodal imaging applications. In practice, it is not uncommon that some FL participants only possess a subset of the complete imaging modalities, posing inter-modal heterogeneity as a challenge to effectively training a global model on all participants' data. In addition, each participant would expect to obtain a personalized model tailored for its local data characteristics from the FL in such a scenario. In this work, we propose a new FL framework with federated modality-specific encoders and multimodal anchors (FedMEMA) to simultaneously address the two concurrent issues. Above all, FedMEMA employs an exclusive encoder for each modality to account for the inter-modal heterogeneity in the first place. In the meantime, while the encoders are shared by the participants, the decoders are personalized to meet individual needs. Specifically, a server with full-modal data employs a fusion decoder to aggregate and fuse representations from all modality-specific encoders, thus bridging the modalities to optimize the encoders via backpropagation reversely. Meanwhile, multiple anchors are extracted from the fused multimodal representations and distributed to the clients in addition to the encoder parameters. On the other end, the clients with incomplete modalities calibrate their missing-modal representations toward the global full-modal anchors via scaled dot-product cross-attention, making up the information loss due to absent modalities while adapting the representations of present ones. FedMEMA is validated on the BraTS 2020 benchmark for multimodal brain tumor segmentation. Results show that it outperforms various up-to-date methods for multimodal and personalized FL and that its novel designs are effective. Our code is available.
Paper Structure (25 sections, 1 equation, 4 figures, 8 tables, 1 algorithm)

This paper contains 25 sections, 1 equation, 4 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Brain Tumor Segmentation with Multimodal MRI:
  4. Multimodal FL with Data Heterogeneity:
  5. Method
  6. Problem Definition:
  7. Framework Overview:
  8. FL with Modality-specific Encoders:
  9. Multi-Anchor Multimodal Representation:
  10. Localized Adaptive Calibration via Cross-Attention:
  11. Experiments and Results
  12. Dataset and Experimental Settings:
  13. Implementation:
  14. Validating Designs for Multi-Anchor Multimodal Representations:
  15. Comparison with Baseline and State-of-the-Art (SOTA) Methods:
  16. ...and 10 more sections

Figures (4)

  • Figure 1: (a) Example images of the four modalities in BraTS 2020 menze2014multimodal. (b) Histograms of the brain and tumor pixels for the four modalities.
  • Figure 2: Overview of the proposed FedMEMA framework. FedMEMA employs a federated encoder exclusive for each modality followed by a server-end multimodal fusion decoder. Meanwhile, personalized decoders are used for the clients to allow simultaneous personalization. In addition, multi-anchor multimodal representations are extracted from the server and distributed to the clients for localized adaptive calibration of modality-specific features via cross-attention. ED: edema, ET: enhancing tumor, NET: necrotic and non-enhancing tumor core, and BG: background.
  • Figure 3: Example segmentation results in experimental setting 1 for a subject in the test set. FLAIR, T1c, T1, and T2 indicate the clients with the corresponding data modalities, and "S" indicates the server. Red: necrotic and non-enhancing tumor core, blue: enhancing tumor, and green: edema.
  • Figure S1: Client-wise cross-attention, i.e., $F_lA^T_l$ in Eq. (2). The first column shows the subregional tumor mask.