WarmFed: Federated Learning with Warm-Start for Globalization and Personalization Via Personalized Diffusion Models

TL;DR

This work addresses privacy-preserving Federated Learning by balancing globalization and personalization without sacrificing client data privacy. It introduces WarmFed, which starts from a pre-trained initialization and creates client-specific warm-start diffusion models via LoRA-fine-tuning, transmitting only compact parameter matrices. At the server, synthetic data generated from these models supports globalization through targeted fine-tuning, while Dynamic Self-Distillation selects personalized knowledge to distill into the global model, enhancing personalization. The approach delivers strong performance in one-shot and five-round communications across diverse datasets with low transmission costs and robust privacy, offering a practical pathway to jointly global and personalized FL.

Abstract

Federated Learning (FL) stands as a prominent distributed learning paradigm among multiple clients to achieve a unified global model without privacy leakage. In contrast to FL, Personalized federated learning aims at serving for each client in achieving persoanlized model. However, previous FL frameworks have grappled with a dilemma: the choice between developing a singular global model at the server to bolster globalization or nurturing personalized model at the client to accommodate personalization. Instead of making trade-offs, this paper commences its discourse from the pre-trained initialization, obtaining resilient global information and facilitating the development of both global and personalized models. Specifically, we propose a novel method called WarmFed to achieve this. WarmFed customizes Warm-start through personalized diffusion models, which are generated by local efficient fine-tunining (LoRA). Building upon the Warm-Start, we advance a server-side fine-tuning strategy to derive the global model, and propose a dynamic self-distillation (DSD) to procure more resilient personalized models simultaneously. Comprehensive experiments underscore the substantial gains of our approach across both global and personalized models, achieved within just one-shot and five communication(s).

Figures (6)

• Figure 1: (a) The performance of personalized model. (b) The details on one client personalization.
• Figure 2: Pipeline of WarmFed, which consists of three stages: (1)For warm-start stage, we fine-tune SD through LoRA and send the fine-tuned parameter matrixes to the server for warm-start. (2)For globalization stage, we fine-tune the local models and aggregated model with synthetic data to achieve final global model. (3)For personalization stage, the personlized model is obtained by dynamic self-distillation, which is employed to select personlized knowledge for self-distillation.
• Figure 3: (a) The feature distribution comparison of synthetic data with private data (Office-Caltech 10). (b) One-shot performance under different amounts of synthetic data relative to the amount of private data (Office-Caltech 10).
• Figure 4: Visualization on synthetic and retrieved real data.
• Figure 5: The performance comparison of personalized model among DSD, pFedSD (SD), Warm-Start.