Secure Federated Learning Across Heterogeneous Cloud and High-Performance Computing Resources -- A Case Study on Federated Fine-tuning of LLaMA 2

TL;DR

The design of the Advanced Privacy-Preserving Federated Learning (APPFL) framework is elaborated on, which streamlines end-to-end secure and reliable federated learning experiments across cloud computing facilities and high-performance computing resources by leveraging Globus Compute, a distributed function as a service platform, and Amazon Web Services.

Abstract

Federated learning enables multiple data owners to collaboratively train robust machine learning models without transferring large or sensitive local datasets by only sharing the parameters of the locally trained models. In this paper, we elaborate on the design of our Advanced Privacy-Preserving Federated Learning (APPFL) framework, which streamlines end-to-end secure and reliable federated learning experiments across cloud computing facilities and high-performance computing resources by leveraging Globus Compute, a distributed function as a service platform, and Amazon Web Services. We further demonstrate the use case of APPFL in fine-tuning a LLaMA 2 7B model using several cloud resources and supercomputers.

Figures (4)

• Figure 1: Overview of the federated learning process using the APPFL framework.
• Figure 2: Modular design of the APPFL framework.
• Figure 3: Overview of the federated large language model fine-tuning experiments among four heterogeneous clients on HPC nodes and cloud.
• Figure 4: Local data distributions among four clients for the SuperGLUE datasets, where different colors indicate samples with different labels.