An Upload-Efficient Scheme for Transferring Knowledge From a Server-Side Pre-trained Generator to Clients in Heterogeneous Federated Learning

TL;DR

HtFL enables task-specific learning across heterogeneous clients but faces data/model heterogeneity and privacy-barrier challenges. FedKTL addresses this by using a server-side public generator to produce task-relevant image–vector prototypes and an ETF-based proxy classifier to create unbiased prototypes, then aligning these prototypes with the generator's latent space via a trainable transformer and a domain-alignment objective. The transferred global knowledge is injected through a lightweight local task, enabling heterogeneous clients to benefit from shared information with minimal upload. Empirically, FedKTL outperforms seven state-of-the-art baselines on four datasets across 14 models, with up to 7.31% accuracy gains, while maintaining low upload costs and compatibility with multiple pre-trained generators, highlighting its practicality for cloud-edge heterogeneous FL deployments.

Abstract

Heterogeneous Federated Learning (HtFL) enables task-specific knowledge sharing among clients with different model architectures while preserving privacy. Despite recent research progress, transferring knowledge in HtFL is still difficult due to data and model heterogeneity. To tackle this, we introduce a public pre-trained generator (e.g., StyleGAN or Stable Diffusion) as the bridge and propose a new upload-efficient knowledge transfer scheme called Federated Knowledge-Transfer-Loop (FedKTL). It can produce task-related prototypical image-vector pairs via the generator's inference on the server. With these pairs, each client can transfer common knowledge from the generator to its local model through an additional supervised local task. We conduct extensive experiments on four datasets under two types of data heterogeneity with 14 heterogeneous models, including CNNs and ViTs. Results show that our FedKTL surpasses seven state-of-the-art methods by up to 7.31%. Moreover, our knowledge transfer scheme is applicable in cloud-edge scenarios with only one edge client. Code: https://github.com/TsingZ0/FedKTL

Figures (12)

• Figure 1: The images ($64\times 64$) generated by StyleGAN-XL sauer2022StyleGAN with different kinds of inputs. "vecs" is short for vectors.
• Figure 2: An illustration of the generating process (from right to left) when utilizing StyleGAN-XL as an example. The solid border of $G_s$ and $G_m$ means "with frozen parameters".
• Figure 3: An example of our FedKTL for a 3-class classification task. (a) Rounded and slender rectangles denote models and representations, respectively; dash-dotted and solid borders denote updating and frozen components, respectively; the segmented circle represents the ETF classifier. (b) The feature transformer ($F$) contains two FC layers and one Batch Normalization ioffe2015batch (BN) layer. (c) An example of the domain alignment step with $K=2$ and $H=3$; one cluster represents one class. Best viewed in color.
• Figure 4: (a): Four images (one image per class) on client #1. (b), (c), (d), and (e): The images generated by different StyleGAN3s correspond to the aforementioned four classes.
• Figure 5: The images generated by StyleGAN-XL correspond to four classes at different iterations.