Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CAR-MFL: Cross-Modal Augmentation by Retrieval for Multimodal Federated Learning with Missing Modalities

Pranav Poudel, Prashant Shrestha, Sanskar Amgain, Yash Raj Shrestha, Prashnna Gyawali, Binod Bhattarai

TL;DR

This work tackles missing modalities in multimodal federated learning for healthcare while preserving data privacy. It introduces CAR-MFL, a retrieval-based cross-modal augmentation method that augments unimodal clients with complementary modalities drawn from a small public multimodal dataset via intra-modal retrieval and label-aware refinement. During federated training, a fixed-constraint weight adjustment is applied to the complementary encoders to mitigate label noise, and augmentations are performed locally to avoid sharing pairing information. Empirical results on chest X-ray benchmarks show CAR-MFL consistently outperforms baselines such as mFedAvgP and CreamFL across both homogeneous and heterogeneous partitions, with robustness to limited public data and improved handling of rare pathologies. This approach enables practical deployment of multimodal FL in healthcare with missing modalities without requiring large public datasets or synchronized representations.

Abstract

Multimodal AI has demonstrated superior performance over unimodal approaches by leveraging diverse data sources for more comprehensive analysis. However, applying this effectiveness in healthcare is challenging due to the limited availability of public datasets. Federated learning presents an exciting solution, allowing the use of extensive databases from hospitals and health centers without centralizing sensitive data, thus maintaining privacy and security. Yet, research in multimodal federated learning, particularly in scenarios with missing modalities a common issue in healthcare datasets remains scarce, highlighting a critical area for future exploration. Toward this, we propose a novel method for multimodal federated learning with missing modalities. Our contribution lies in a novel cross-modal data augmentation by retrieval, leveraging the small publicly available dataset to fill the missing modalities in the clients. Our method learns the parameters in a federated manner, ensuring privacy protection and improving performance in multiple challenging multimodal benchmarks in the medical domain, surpassing several competitive baselines. Code Available: https://github.com/bhattarailab/CAR-MFL

CAR-MFL: Cross-Modal Augmentation by Retrieval for Multimodal Federated Learning with Missing Modalities

TL;DR

This work tackles missing modalities in multimodal federated learning for healthcare while preserving data privacy. It introduces CAR-MFL, a retrieval-based cross-modal augmentation method that augments unimodal clients with complementary modalities drawn from a small public multimodal dataset via intra-modal retrieval and label-aware refinement. During federated training, a fixed-constraint weight adjustment is applied to the complementary encoders to mitigate label noise, and augmentations are performed locally to avoid sharing pairing information. Empirical results on chest X-ray benchmarks show CAR-MFL consistently outperforms baselines such as mFedAvgP and CreamFL across both homogeneous and heterogeneous partitions, with robustness to limited public data and improved handling of rare pathologies. This approach enables practical deployment of multimodal FL in healthcare with missing modalities without requiring large public datasets or synchronized representations.

Abstract

Multimodal AI has demonstrated superior performance over unimodal approaches by leveraging diverse data sources for more comprehensive analysis. However, applying this effectiveness in healthcare is challenging due to the limited availability of public datasets. Federated learning presents an exciting solution, allowing the use of extensive databases from hospitals and health centers without centralizing sensitive data, thus maintaining privacy and security. Yet, research in multimodal federated learning, particularly in scenarios with missing modalities a common issue in healthcare datasets remains scarce, highlighting a critical area for future exploration. Toward this, we propose a novel method for multimodal federated learning with missing modalities. Our contribution lies in a novel cross-modal data augmentation by retrieval, leveraging the small publicly available dataset to fill the missing modalities in the clients. Our method learns the parameters in a federated manner, ensuring privacy protection and improving performance in multiple challenging multimodal benchmarks in the medical domain, surpassing several competitive baselines. Code Available: https://github.com/bhattarailab/CAR-MFL
Paper Structure (8 sections, 6 equations, 7 figures, 5 tables)

This paper contains 8 sections, 6 equations, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Experiments and Results
  4. Quantitative Results
  5. Qualitative Results
  6. Conclusion
  7. Acknowledgments.
  8. Supplementary Materials

Figures (7)

  • Figure 1: Illustration of CAR-MFL. (a) Multimodal client with access to multimodal data. (b) Multimodal federated system with missing modality. (c) Image client with only image samples; missing text modality is retrieved via our Cross-Modal Augmentation module. (d) Cross-Modal augmentation procedure for a query image (yellow): Most relevant image from public data is retrieved based on distance in feature space and label similarity. Then, the associated text of the retrieved image is paired with the query image forming a paired input.
  • Figure 2: Comparison between mFedAvgP and CAR-MFL on rare pathologies.
  • Figure 3: (a) Study of model AUC on varying public data size. (b) Distribution of weight values of classification layer across image and text features.
  • Figure 4: Qualitative Analysis of retrieved samples across different training rounds. Column 1 contains a paired image text sample. The first row displays retrieved text reports from the public dataset when the text modality is missing. The second row displays retrieved images when the image modality is missing.
  • Figure 5: Label distribution of different datasets in our setup.
  • ...and 2 more figures