FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Models

TL;DR

The paper proposes FATE-LLM, an industrial-grade federated learning framework for large language models that integrates FedLLM paradigms, parameter-efficient fine-tuning, and privacy/IP protections to enable collaborative training across organizations with varying resources. It details the system design, architecture, and roadmap built on the FATE ecosystem, along with experiments using ChatGLM-6B and LoRA/P-Tuning-v2 to demonstrate improved performance over local fine-tuning and substantial reductions in communication cost versus full fine-tuning. The key contributions include a modular FedLLM stack (FedHomoLLM, FedHeteroLLM, FedCoLLM, FedOST), privacy/IP protection via FedIPR, and practical evaluation showing the viability of federated LLM training for industry. The work aims to broaden access to powerful LLMs for SMEs while preserving data privacy and IP, and it provides open-source tooling to accelerate adoption and further research.

Abstract

Large Language Models (LLMs), such as ChatGPT, LLaMA, GLM, and PaLM, have exhibited remarkable performances across various tasks in recent years. However, LLMs face two main challenges in real-world applications. One challenge is that training LLMs consumes vast computing resources, preventing LLMs from being adopted by small and medium-sized enterprises with limited computing resources. Another is that training LLM requires a large amount of high-quality data, which are often scattered among enterprises. To address these challenges, we propose FATE-LLM, an industrial-grade federated learning framework for large language models. FATE-LLM (1) facilitates federated learning for large language models (coined FedLLM); (2) promotes efficient training of FedLLM using parameter-efficient fine-tuning methods; (3) protects the intellectual property of LLMs; (4) preserves data privacy during training and inference through privacy-preserving mechanisms. We release the code of FATE-LLM at https://github.com/FederatedAI/FATE-LLM to facilitate the research of FedLLM and enable a broad range of industrial applications.

Figures (7)

• Figure 1: Large Language Models are federated on FATE.
• Figure 2: Components of the FATE-LLM system.
• Figure 3: FATE-LLM Trainers. FATE-LLM offers four trainers for four different federated LLM learning scenarios.
• Figure 4: FedIPR!li2022fedipr. Private watermarks are generated and embedded into the trainable parameters (i.e., adaptors or prompts) of local large language models. Then, trainable parameters are aggregated through FedAvg.
• Figure 5: Architecture of the FATE-LLM system.