Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Recent Advances in Federated Learning Driven Large Language Models: A Survey on Architecture, Performance, and Security

Youyang Qu, Ming Liu, Tianqing Zhu, Longxiang Gao, Shui Yu, Wanlei Zhou

TL;DR

The paper surveys Federated Learning (FL) approaches for training Large Language Models (LLMs) across decentralized data sources to preserve privacy and reduce communication. It surveys architectures, performance optimizations, and security considerations, including machine unlearning as a mechanism to comply with regulations. It reviews methods such as secure aggregation, differential privacy, prompt tuning, hierarchical aggregation, and model splitting, and discusses their trade-offs and practical implications via case studies. The work identifies key open challenges and directions for building secure, scalable, and adaptable FL-LMM systems with real-world impact in sensitive domains.

Abstract

Federated Learning (FL) offers a promising paradigm for training Large Language Models (LLMs) in a decentralized manner while preserving data privacy and minimizing communication overhead. This survey examines recent advancements in FL-driven LLMs, with a particular emphasis on architectural designs, performance optimization, and security concerns, including the emerging area of machine unlearning. In this context, machine unlearning refers to the systematic removal of specific data contributions from trained models to comply with privacy regulations such as the Right to be Forgotten. We review a range of strategies enabling unlearning in federated LLMs, including perturbation-based methods, model decomposition, and incremental retraining, while evaluating their trade-offs in terms of efficiency, privacy guarantees, and model utility. Through selected case studies and empirical evaluations, we analyze how these methods perform in practical FL scenarios. This survey identifies critical research directions toward developing secure, adaptable, and high-performing federated LLM systems for real-world deployment.

Recent Advances in Federated Learning Driven Large Language Models: A Survey on Architecture, Performance, and Security

TL;DR

The paper surveys Federated Learning (FL) approaches for training Large Language Models (LLMs) across decentralized data sources to preserve privacy and reduce communication. It surveys architectures, performance optimizations, and security considerations, including machine unlearning as a mechanism to comply with regulations. It reviews methods such as secure aggregation, differential privacy, prompt tuning, hierarchical aggregation, and model splitting, and discusses their trade-offs and practical implications via case studies. The work identifies key open challenges and directions for building secure, scalable, and adaptable FL-LMM systems with real-world impact in sensitive domains.

Abstract

Federated Learning (FL) offers a promising paradigm for training Large Language Models (LLMs) in a decentralized manner while preserving data privacy and minimizing communication overhead. This survey examines recent advancements in FL-driven LLMs, with a particular emphasis on architectural designs, performance optimization, and security concerns, including the emerging area of machine unlearning. In this context, machine unlearning refers to the systematic removal of specific data contributions from trained models to comply with privacy regulations such as the Right to be Forgotten. We review a range of strategies enabling unlearning in federated LLMs, including perturbation-based methods, model decomposition, and incremental retraining, while evaluating their trade-offs in terms of efficiency, privacy guarantees, and model utility. Through selected case studies and empirical evaluations, we analyze how these methods perform in practical FL scenarios. This survey identifies critical research directions toward developing secure, adaptable, and high-performing federated LLM systems for real-world deployment.
Paper Structure (18 sections, 6 figures, 5 tables)

This paper contains 18 sections, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Foundations of Federated Learning and Large Language Models
  3. Federated Learning: Principles & Motivations
  4. Fundamentals of Large Language Models
  5. Integration of Federated Learning and LLMs
  6. Federated Large Language Models
  7. Federated LLM architectures
  8. Efficient Fine-tuning of LLM
  9. Pre-Training of LLM in Federated Learning
  10. Scalable LLM via Federated Learning
  11. Security and Privacy in Federated LLM
  12. Open Challenges and Future Directions
  13. Open Challenges and Future Directions in LLM Architectures in Federated Settings
  14. Open Challenges and Future Directions in Efficient Fine-Tuning of Federated LLM
  15. Open Challenges and Future Directions in Pre-Training of LLM in Federated Learning
  16. ...and 3 more sections

Figures (6)

  • Figure 1: Organization of this survey
  • Figure 2: General architecture of Federated Learning for Large Language Models
  • Figure 3: Fine-tuning of Federated LLMs for Swarm Intelligence
  • Figure 4: Pre-training of Federated LLMs for Swarm Intelligence
  • Figure 5: Scalable Federated LLMs for Swarm Intelligence
  • ...and 1 more figures