A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learning
Jieming Bian, Yuanzhe Peng, Lei Wang, Yin Huang, Jie Xu
TL;DR
Foundation models require substantial adaptation for downstream tasks, motivating parameter-efficient fine-tuning (PEFT) to reduce updates. This survey systematically analyzes how PEFT methods—categorized as Additive, Selective, and Reparameterized—are integrated with Federated Learning to enable privacy-preserving, scalable fine-tuning across diverse NLP and CV tasks. It synthesizes methodologies, datasets, and model families, discusses challenges such as data heterogeneity and communication overhead, and highlights approaches (e.g., adapters, prompts, and LoRA-based reparameterizations) that mitigate FL-specific constraints. The paper identifies key gaps, including scaling to ultra-large foundation models, theoretical convergence analyses, and sustainable, energy-aware training, offering a roadmap for future research and practice in FL-PEFT. In short, FL-PEFT provides a practical framework for efficient, private, and adaptable deployment of foundation models in distributed environments.
Abstract
Foundation models have revolutionized artificial intelligence by providing robust, versatile architectures pre-trained on large-scale datasets. However, adapting these massive models to specific downstream tasks requires fine-tuning, which can be prohibitively expensive in computational resources. Parameter-Efficient Fine-Tuning (PEFT) methods address this challenge by selectively updating only a small subset of parameters. Meanwhile, Federated Learning (FL) enables collaborative model training across distributed clients without sharing raw data, making it ideal for privacy-sensitive applications. This survey provides a comprehensive review of the integration of PEFT techniques within federated learning environments. We systematically categorize existing approaches into three main groups: Additive PEFT (which introduces new trainable parameters), Selective PEFT (which fine-tunes only subsets of existing parameters), and Reparameterized PEFT (which transforms model architectures to enable efficient updates). For each category, we analyze how these methods address the unique challenges of federated settings, including data heterogeneity, communication efficiency, computational constraints, and privacy concerns. We further organize the literature based on application domains, covering both natural language processing and computer vision tasks. Finally, we discuss promising research directions, including scaling to larger foundation models, theoretical analysis of federated PEFT methods, and sustainable approaches for resource-constrained environments.