Synergizing Foundation Models and Federated Learning: A Survey

TL;DR

The paper tackles the challenge of harmonizing Foundation Models with Federated Learning to enable privacy-preserving, data-efficient adaptation of large pre-trained models across diverse domains. It proposes a comprehensive three-dimensional taxonomy—efficiency, adaptability, and trustworthiness—and surveys methods spanning FedPEFT, model compression, distillation, and zeroth-order optimization, complemented by an analysis of libraries, benchmarks, and applications. The work highlights practical implications for IP protection, privacy, and robustness while detailing domain- and client-centric adaptation strategies and system-level tooling for FedFM deployment. Overall, the survey provides a structured blueprint for researchers and practitioners to design, evaluate, and deploy FedFM systems in privacy-conscious settings with a view toward multimodal, continual, and AI-generated content-enabled futures.

Abstract

Over the past few years, the landscape of Artificial Intelligence (AI) has been reshaped by the emergence of Foundation Models (FMs). Pre-trained on massive datasets, these models exhibit exceptional performance across diverse downstream tasks through adaptation techniques like fine-tuning and prompt learning. More recently, the synergy of FMs and Federated Learning (FL) has emerged as a promising paradigm, often termed Federated Foundation Models (FedFM), allowing for collaborative model adaptation while preserving data privacy. This survey paper provides a systematic review of the current state of the art in FedFM, offering insights and guidance into the evolving landscape. Specifically, we present a comprehensive multi-tiered taxonomy based on three major dimensions, namely efficiency, adaptability, and trustworthiness. To facilitate practical implementation and experimental research, we undertake a thorough review of existing libraries and benchmarks. Furthermore, we discuss the diverse real-world applications of this paradigm across multiple domains. Finally, we outline promising research directions to foster future advancements in FedFM. Overall, this survey serves as a resource for researchers and practitioners, offering a thorough understanding of FedFM's role in revolutionizing privacy-preserving AI and pointing toward future innovations in this promising area. A periodically updated paper collection on FM-FL is available at https://github.com/lishenghui/awesome-fm-fl.

Figures (3)

• Figure 1: Taxonomy of research in foundation models with federated learning.
• Figure 2: Taxonomy of Federated Parameter-Efficient Fine-Tuning (FedPEFT). Apart from efficiency, some methods also account for other considerations, such as data and resource heterogeneity challenges that are identified in Section \ref{['sec:taskchallenges']} and black-box tuning (see Section \ref{['sec:sp']}).
• Figure 3: Illustration of undesired model behaviors caused by two representative targeted attacks. (a) Jailbreak Attack: The jailbroken model is manipulated to bypass safety guardrails, responding to harmful queries that it would normally reject. (b) Backdoor Attack: A maliciously injected trigger (e.g., "Wow!") flips a sentiment classification result