Federated Customization of Large Models: Approaches, Experiments, and Insights

TL;DR

Federated customization of large language models addresses privacy and regulatory constraints by keeping client data local while sharing model updates. The paper surveys six customization techniques and shows how each can be adapted to federated learning, then reports a first experimental study of federated prefix-tuning that achieves performance close to centralized baselines and competitive results against other federated methods. Through systematic experiments on table-to-text tasks (E2E and DART), it analyzes the effects of model scale, client count, and non-IID data, highlighting the robustness and efficiency of federated prefix-tuning. The work provides practical guidance and identifies future directions toward multi-task/multi-modal settings, privacy-enhanced protocols, and lightweight federated customization for resource-constrained deployments.

Abstract

In this article, we explore federated customization of large models and highlight the key challenges it poses within the federated learning framework. We review several popular large model customization techniques, including full fine-tuning, efficient fine-tuning, prompt engineering, prefix-tuning, knowledge distillation, and retrieval-augmented generation. Then, we discuss how these techniques can be implemented within the federated learning framework. Moreover, we conduct experiments on federated prefix-tuning, which, to the best of our knowledge, is the first trial to apply prefix-tuning in the federated learning setting. The conducted experiments validate its feasibility with performance close to centralized approaches. Further comparison with three other federated customization methods demonstrated its competitive performance, satisfactory efficiency, and consistent robustness.

Figures (1)

• Figure 1: Frameworks of different federated LM customization methods.