MQFL-FHE: Multimodal Quantum Federated Learning Framework with Fully Homomorphic Encryption

TL;DR

This work addresses privacy and efficiency challenges in federated learning by integrating fully homomorphic encryption (FHE) with quantum computing to counteract aggregation-induced degradation, particularly in multimodal settings. It introduces MQFL-FHE and the Multimodal Quantum Mixture of Experts (MQMoE-FL-FHE) framework, enabling secure, quantum-assisted, multimodal learning across heterogeneous data sources. Through theoretical analysis and empirical evaluation on datasets spanning images, DNA sequences, and medical imaging, the authors show that quantum enhancements can improve representational generalization and mitigate FHE-related performance losses, while highlighting trade-offs in computational overhead. The study demonstrates the potential of quantum interventions to advance privacy-preserving FL for complex, multimodal applications with underrepresented classes.

Abstract

The integration of fully homomorphic encryption (FHE) in federated learning (FL) has led to significant advances in data privacy. However, during the aggregation phase, it often results in performance degradation of the aggregated model, hindering the development of robust representational generalization. In this work, we propose a novel multimodal quantum federated learning framework that utilizes quantum computing to counteract the performance drop resulting from FHE. For the first time in FL, our framework combines a multimodal quantum mixture of experts (MQMoE) model with FHE, incorporating multimodal datasets for enriched representation and task-specific learning. Our MQMoE framework enhances performance on multimodal datasets and combined genomics and brain MRI scans, especially for underrepresented categories. Our results also demonstrate that the quantum-enhanced approach mitigates the performance degradation associated with FHE and improves classification accuracy across diverse datasets, validating the potential of quantum interventions in enhancing privacy in FL.

Figures (6)

• Figure 1: Overview of our novel contributions.
• Figure 2: Overview of the MQFL-FHE framework. Each client (e.g., client 1, client 2, etc.) trains a local model on its private dataset, encrypts the model weights using the CKKS homomorphic encryption scheme, and sends the encrypted local model $\mathbf{w}_k^{enc}$ to the central server. The global server securely aggregates the encrypted local models using a weighted sum based on client data contributions. The aggregated model is then decrypted, optimized, and distributed back to all clients as the updated global model $\mathbf{w}_g$. A single key setup is used for both encryption and decryption, ensuring secure communication throughout the process.
• Figure 3: The model workflow follows a MQMoE approach. MRI and DNA data are processed through distinct classical layers, where MRI inputs pass through convolutional and pooling layers, while DNA inputs are processed through linear layers. After classical preprocessing, both inputs are passed through quantum layers outlined in the figure, representing QNNs, with 6-qubit quantum layers for both MRI & DNA to create two quantum expert representational vectors. These quantum layers serve as specialized experts to extract complex feature representations. The outputs from the quantum experts are combined through feature concatenation, enhanced by a multi-head attention (MHA) mechanism to capture key features from the input data. For experts, the MHA+FC layer+Softmax is the gating network. Combined with the weighted sum, this integrates the quantum expert outputs, leading to their respective output layers. This process optimizes the final predictions for both MRI and DNA output layers.
• Figure 4: ROC curves for DNA and MRI datasets under FL with and without quantum enhancements. Panels (a) and (b) show the ROC curves for the DNA dataset under FL-FHE and QFL-FHE setups, respectively, highlighting the performance across various classes. Panels (c) and (d) show the ROC curves for the MRI dataset under FL-FHE and QFL-FHE setups, respectively. Each panel demonstrates the true positive rate against the false positive rate for each class, including micro-average and macro-average ROC curves, illustrating the models' discriminative ability in a privacy-preserving FL context.
• Figure 5: Confusion matrices for DNA and MRI datasets under FL with and without quantum enhancements. The first two matrices on the left illustrate the classification performance for the DNA dataset using FL+FHE and QFL+FHE settings, respectively. These matrices show the distribution of true labels versus predicted labels, with the intensity of the colors indicating the proportion of predictions. The right two matrices focus on the MRI dataset under the same setting, FL+FHE, and QFL+FHE. Each matrix highlights how effectively each model categorizes different tumor types, such as glioma, meningioma, no tumor, and pituitary, with brighter colors representing higher frequencies of predictions.