Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Large Language Models: Feasibility, Robustness, Security and Future Directions

Wenhao Jiang, Yuchuan Luo, Guilin Deng, Silong Chen, Xu Yang, Shihong Wu, Xinwen Gao, Lin Liu, Shaojing Fu

TL;DR

This paper surveys Federated Large Language Models (FLLM), focusing on feasibility, robustness, security, and future directions to enable privacy-preserving training on distributed data silos. It argues that full-parameter fine-tuning is generally infeasible at scale and foregrounds Federated Parameter-Efficient Fine-Tuning (FedPEFT) with methods like LoRA, Adapter, and Prompt Tuning as practical alternatives. The analysis covers robustness against resource, data, and task heterogeneity, and details privacy and security threats (e.g., MIA, DRA, jailbreaking, prompt injection, LTDL) along with defense mechanisms such as robust aggregation and differential privacy. It also outlines promising directions, including few-shot learning, federated unlearning, and IP protection, highlighting the need for further work to build robust, secure, and scalable FLLM systems with real-world impact.

Abstract

The integration of Large Language Models (LLMs) and Federated Learning (FL) presents a promising solution for joint training on distributed data while preserving privacy and addressing data silo issues. However, this emerging field, known as Federated Large Language Models (FLLM), faces significant challenges, including communication and computation overheads, heterogeneity, privacy and security concerns. Current research has primarily focused on the feasibility of FLLM, but future trends are expected to emphasize enhancing system robustness and security. This paper provides a comprehensive review of the latest advancements in FLLM, examining challenges from four critical perspectives: feasibility, robustness, security, and future directions. We present an exhaustive survey of existing studies on FLLM feasibility, introduce methods to enhance robustness in the face of resource, data, and task heterogeneity, and analyze novel risks associated with this integration, including privacy threats and security challenges. We also review the latest developments in defense mechanisms and explore promising future research directions, such as few-shot learning, machine unlearning, and IP protection. This survey highlights the pressing need for further research to enhance system robustness and security while addressing the unique challenges posed by the integration of FL and LLM.

Federated Large Language Models: Feasibility, Robustness, Security and Future Directions

TL;DR

This paper surveys Federated Large Language Models (FLLM), focusing on feasibility, robustness, security, and future directions to enable privacy-preserving training on distributed data silos. It argues that full-parameter fine-tuning is generally infeasible at scale and foregrounds Federated Parameter-Efficient Fine-Tuning (FedPEFT) with methods like LoRA, Adapter, and Prompt Tuning as practical alternatives. The analysis covers robustness against resource, data, and task heterogeneity, and details privacy and security threats (e.g., MIA, DRA, jailbreaking, prompt injection, LTDL) along with defense mechanisms such as robust aggregation and differential privacy. It also outlines promising directions, including few-shot learning, federated unlearning, and IP protection, highlighting the need for further work to build robust, secure, and scalable FLLM systems with real-world impact.

Abstract

The integration of Large Language Models (LLMs) and Federated Learning (FL) presents a promising solution for joint training on distributed data while preserving privacy and addressing data silo issues. However, this emerging field, known as Federated Large Language Models (FLLM), faces significant challenges, including communication and computation overheads, heterogeneity, privacy and security concerns. Current research has primarily focused on the feasibility of FLLM, but future trends are expected to emphasize enhancing system robustness and security. This paper provides a comprehensive review of the latest advancements in FLLM, examining challenges from four critical perspectives: feasibility, robustness, security, and future directions. We present an exhaustive survey of existing studies on FLLM feasibility, introduce methods to enhance robustness in the face of resource, data, and task heterogeneity, and analyze novel risks associated with this integration, including privacy threats and security challenges. We also review the latest developments in defense mechanisms and explore promising future research directions, such as few-shot learning, machine unlearning, and IP protection. This survey highlights the pressing need for further research to enhance system robustness and security while addressing the unique challenges posed by the integration of FL and LLM.
Paper Structure (28 sections, 7 figures, 5 tables)

This paper contains 28 sections, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Feasibility of FLLM
  3. Full-parameter Fine-tuning
  4. Parameter-Efficient Fine-Tuning
  5. Prompt tuning
  6. Other Methods
  7. Robustness of FLLM
  8. Resource Heterogeneity
  9. Data Heterogeneity
  10. Task Heterogeneity
  11. Security of FLLM
  12. Privacy Leakage and Defenses
  13. Membership Inference Attack
  14. Data Reconstruction Attack
  15. Jailbreaking Attack
  16. ...and 13 more sections

Figures (7)

  • Figure 1: The framework of this survey. Darker circles indicate a higher number of related studies.
  • Figure 2: The fine-tuning process of full-parameter fine-tuning, PEFT (including partial-parameter fine-tuning, LoRA, Adapter) and prompt tuning. The orange box denotes the trainable parameters.
  • Figure 3: The process of Adapter-based Fine-tuning and LoRA-based Fine-tuning.
  • Figure 4: The framework and workflow of PROMPTFL from guo2023promptfl. Clients locally train a prompt learner with a small number of parameters, without altering the original LLM, and the server aggregates only the updates from the prompt learners.
  • Figure 5: FlexLoRA workflow from bai2024federated. The server aggregates the full-size LoRA after multiplication, not the individual matrices A and B. The global full-size LoRA is then decomposed into smaller matrices of varying ranks via SVD and allocated sequentially based on client resources.
  • ...and 2 more figures