BlendFL: Blended Federated Learning for Handling Multimodal Data Heterogeneity

TL;DR

BlendFL addresses multimodal data heterogeneity in collaborative settings by unifying horizontal and vertical federated learning, enabling both unimodal and multimodal predictions without strict data-sharing constraints. It introduces BlendAvg, a performance-driven model aggregation method that prioritizes better-performing local updates using a private validation set, and supports decentralized inference to reduce server reliance. Across real-world clinical data and the S-MNIST benchmark, BlendFL demonstrates superior or on-par performance with centralized learning and faster convergence than traditional FL baselines. The framework shows strong potential for privacy-preserving, multimodal collaboration in healthcare and finance, with future work focusing on incorporating stronger privacy guarantees such as differential privacy.

Abstract

One of the key challenges of collaborative machine learning, without data sharing, is multimodal data heterogeneity in real-world settings. While Federated Learning (FL) enables model training across multiple clients, existing frameworks, such as horizontal and vertical FL, are only effective in `ideal' settings that meet specific assumptions. Hence, they struggle to address scenarios where neither all modalities nor all samples are represented across the participating clients. To address this gap, we propose BlendFL, a novel FL framework that seamlessly blends the principles of horizontal and vertical FL in a synchronized and non-restrictive fashion despite the asymmetry across clients. Specifically, any client within BlendFL can benefit from either of the approaches, or both simultaneously, according to its available dataset. In addition, BlendFL features a decentralized inference mechanism, empowering clients to run collaboratively trained local models using available local data, thereby reducing latency and reliance on central servers for inference. We also introduce BlendAvg, an adaptive global model aggregation strategy that prioritizes collaborative model updates based on each client's performance. We trained and evaluated BlendFL and other state-of-the-art baselines on three classification tasks using a large-scale real-world multimodal medical dataset and a popular multimodal benchmark. Our results highlight BlendFL's superior performance for both multimodal and unimodal classification. Ablation studies demonstrate BlendFL's faster convergence compared to traditional approaches, accelerating collaborative learning. Overall, in our study we highlight the potential of BlendFL for handling multimodal data heterogeneity for collaborative learning in real-world settings where data privacy is crucial, such as in healthcare and finance.

Figures (4)

• Figure 1: BlendFL architecture. We showcase the BlendFL architecture with 3 hospitals: 1 has multimodal data (client 1), while the other 2 hospitals have only unimodal data (clients 2 and 3). Each party $i$ has a specific dataset ($D^i_m$), for a specific modality $m$, where $m \in \{A, B\}$. Note that data can be fragmented (different modalities for the same patient are in different hospitals), partial (only one modality exists for a patient), or paired (both modalities are present in the same client). The dashed and dotted arrows indicate communication between models, either locally or between the hospitals and the server, as detailed in the legend. For clarity, we omitted $f_A$ and $f_B$, which are the feature extractors corresponding to each $g_A$ and $g_B$ classifier. Note that the arrows represent the data flow in one direction, to train the global models $g^{blended}_A$, $g^{blended}_B$, and $g^{blended}_M$. The grey dashed arrows represent the distribution of the global models back to the clients after every epoch.
• Figure 2: Model Convergence. Comparison of rounds needed for model convergence (to reach the target 0.98 AUROC) for the federated model update strategies BlendAvg and FedAvg on the S-MNIST dataset.
• Figure 3: Data Distribution. Performance comparison of BlendFL and FL baselines for different data distribution ratios (paired/partial) on the S-MNIST dataset.
• Figure 4: Number of clients. Performance comparison of BlendFL and FL baselines for varying numbers of clients on the S-MNIST dataset.