OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning

TL;DR

OpenFedLLM presents an integrated framework for privacy-preserving LLM training via federated learning, combining FedIT and FedVA with seven FL baselines across eight datasets and 30+ metrics. The framework demonstrates that FL consistently outperforms isolated training and, in a finance benchmark, can surpass GPT-4, underscoring the value of decentralized private data. Key contributions include a decoupled OpenFedLLM design, comprehensive empirical studies, and insights into data management, heterogeneous preferences, personalization, robustness, privacy, and efficiency. The work highlights the practical potential of FL for scalable, domain-rich LLM development while identifying important directions for future research and framework evolution.

Abstract

Trained on massive publicly available data, large language models (LLMs) have demonstrated tremendous success across various fields. While more data contributes to better performance, a disconcerting reality is that high-quality public data will be exhausted in a few years. In this paper, we offer a potential next step for contemporary LLMs: collaborative and privacy-preserving LLM training on the underutilized distributed private data via federated learning (FL), where multiple data owners collaboratively train a shared model without transmitting raw data. To achieve this, we build a concise, integrated, and research-friendly framework/codebase, named OpenFedLLM. It covers federated instruction tuning for enhancing instruction-following capability, federated value alignment for aligning with human values, and 7 representative FL algorithms. Besides, OpenFedLLM supports training on diverse domains, where we cover 8 training datasets; and provides comprehensive evaluations, where we cover 30+ evaluation metrics. Through extensive experiments, we observe that all FL algorithms outperform local training on training LLMs, demonstrating a clear performance improvement across a variety of settings. Notably, in a financial benchmark, Llama2-7B fine-tuned by applying any FL algorithm can outperform GPT-4 by a significant margin while the model obtained through individual training cannot, demonstrating strong motivation for clients to participate in FL. The code is available at https://github.com/rui-ye/OpenFedLLM.

Figures (3)

• Figure 1: Overview of our proposed OpenFedLLM framework and one example of experimental results. OpenFedLLM integrates 7 representative federated learning algorithms, federated instruction tuning, and federated value alignment, and supports 8 training datasets and 30+ evaluation metrics. The experiments (right) showcase the results of federated instruction tuning on the financial domain, where we see that FL helps train a better LLM that can outperform GPT-4 and GPT-3.5.
• Figure 2: Overview of federated instruction tuning (FedIT). In FedIT, each client holds multiple data samples, where each sample is an (instruction, ground-truth response) pair. The instruction is usually formatted with a prompt template. During local training, the model is trained to predict the response given the template with the instruction, where the base LLM is frozen while only a few learnable parameters are updated (e.g., using LoRA). During communication, only the set of learnable parameters is communicated and aggregated.
• Figure 3: Overview of federated value alignment (FedVA). In FedVA, each client holds multiple data samples, where each one consists of an instruction, a preferred response, and a dispreferred response. The instruction is usually formatted with a prompt template. During local training, the model is trained to align with the preferred response while keeping away from the dispreferred response, where the base LLM is frozen while only a few learnable parameters are introduced (LoRA). During communication, only the set of learnable parameters is communicated and aggregated.