Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modality

TL;DR

This work tackles the challenge of robust multimodal federated learning when clients may be severely missing one or more modalities, a scenario that causes misalignment from zero-filling during local training. It introduces Multimodal Federated Cross Prototype Learning (MFCPL), which employs complete prototypes projected into a shared space to guide local updates across modality-shared and modality-specific representations. MFCPL comprises three interdependent components—Cross-Modal Prototypes Regularization (CMPR), Cross-Modal Prototypes Contrastive (CMPC), and Cross-Modal Alignment (CMA)—that together enable knowledge transfer from complete prototypes to clients with missing data, while mitigating heterogeneity. Extensive experiments on UCI-HAR, Hateful Memes, and MELD show that MFCPL consistently surpasses state-of-the-art baselines, especially as the missing-rate $q$ increases, underscoring its practical significance for robust, privacy-preserving multimodal learning in real-world heterogeneous settings.

Abstract

Multimodal federated learning (MFL) has emerged as a decentralized machine learning paradigm, allowing multiple clients with different modalities to collaborate on training a global model across diverse data sources without sharing their private data. However, challenges, such as data heterogeneity and severely missing modalities, pose crucial hindrances to the robustness of MFL, significantly impacting the performance of global model. The occurrence of missing modalities in real-world applications, such as autonomous driving, often arises from factors like sensor failures, leading knowledge gaps during the training process. Specifically, the absence of a modality introduces misalignment during the local training phase, stemming from zero-filling in the case of clients with missing modalities. Consequently, achieving robust generalization in global model becomes imperative, especially when dealing with clients that have incomplete data. In this paper, we propose $\textbf{Multimodal Federated Cross Prototype Learning (MFCPL)}$, a novel approach for MFL under severely missing modalities. Our MFCPL leverages the complete prototypes to provide diverse modality knowledge in modality-shared level with the cross-modal regularization and modality-specific level with cross-modal contrastive mechanism. Additionally, our approach introduces the cross-modal alignment to provide regularization for modality-specific features, thereby enhancing the overall performance, particularly in scenarios involving severely missing modalities. Through extensive experiments on three multimodal datasets, we demonstrate the effectiveness of MFCPL in mitigating the challenges of data heterogeneity and severely missing modalities while improving the overall performance and robustness of MFL.

Figures (7)

• Figure 1: Problem Illustration of Multimodal Federated Learning with $M=2$. We present two challenges of MFL which are a) Missing Modality and b) Data Heterogeneity.
• Figure 2: Effect of different missing modality rates $q$ on FedAvg on different datasets.
• Figure 3: Illustration of MFCPL with $M=2$. Different client types upload their local prototypes based on Eq. \ref{['proto_1']} to server using projection head $g_1$. We provide diverse modality knowledge with the complete prototypes from Eq. \ref{['aggproto']} in modality-shared level with $\mathcal{L}_{CMPR}$ from Eq. \ref{['eq:cmpr']} and modality-specific level with $\mathcal{L}_{CMPC}$ from Eq. \ref{['cmpc']} using projection head $g_2$. The alignment loss $\mathcal{L}_{CMA}$ in Eq. \ref{['cma']} is introduced to reduce the negative effect arising from zero-filling of missing modalities and enhance the coherence between projected modality-specific representations of two modalities using projection head $g_2$. In the illustration, the square class represents the positive class, while the triangle class represents the negative class.
• Figure 4: The data distribution of training data for each client with non-IID characteristic. The color bar denotes number of data samples. Each rectangle defines the number of samples in each class of each client.
• Figure 5: Illustrative example of performance versus communication rounds three datasets with $q=0.5$.