Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Capture Global Feature Statistics for One-Shot Federated Learning

Zenghao Guan, Yucan Zhou, Xiaoyan Gu

TL;DR

FedCGS tackles the inefficiency and privacy challenges of traditional FL by capturing global feature statistics from local data via a pre-trained backbone. It enables a training-free global one-shot FL via a Gaussian Naive Bayes head and introduces a personalization pathway where clients download global statistics to regularize local representations. The approach demonstrates strong performance across non-IID settings and feature-shift scenarios, with favorable communication overhead and robust privacy considerations, as validated on multiple vision benchmarks. The work provides practical implications for efficient, robust federated learning with pre-trained models and highlights directions for extending to other tasks.

Abstract

Traditional Federated Learning (FL) necessitates numerous rounds of communication between the server and clients, posing significant challenges including high communication costs, connection drop risks and susceptibility to privacy attacks. One-shot FL has become a compelling learning paradigm to overcome above drawbacks by enabling the training of a global server model via a single communication round. However, existing one-shot FL methods suffer from expensive computation cost on the server or clients and cannot deal with non-IID (Independent and Identically Distributed) data stably and effectively. To address these challenges, this paper proposes FedCGS, a novel Federated learning algorithm that Capture Global feature Statistics leveraging pre-trained models. With global feature statistics, we achieve training-free and heterogeneity-resistant one-shot FL. Furthermore, we extend its application to personalization scenario, where clients only need execute one extra communication round with server to download global statistics. Extensive experimental results demonstrate the effectiveness of our methods across diverse data heterogeneity settings. Code is available at https://github.com/Yuqin-G/FedCGS.

Capture Global Feature Statistics for One-Shot Federated Learning

TL;DR

FedCGS tackles the inefficiency and privacy challenges of traditional FL by capturing global feature statistics from local data via a pre-trained backbone. It enables a training-free global one-shot FL via a Gaussian Naive Bayes head and introduces a personalization pathway where clients download global statistics to regularize local representations. The approach demonstrates strong performance across non-IID settings and feature-shift scenarios, with favorable communication overhead and robust privacy considerations, as validated on multiple vision benchmarks. The work provides practical implications for efficient, robust federated learning with pre-trained models and highlights directions for extending to other tasks.

Abstract

Traditional Federated Learning (FL) necessitates numerous rounds of communication between the server and clients, posing significant challenges including high communication costs, connection drop risks and susceptibility to privacy attacks. One-shot FL has become a compelling learning paradigm to overcome above drawbacks by enabling the training of a global server model via a single communication round. However, existing one-shot FL methods suffer from expensive computation cost on the server or clients and cannot deal with non-IID (Independent and Identically Distributed) data stably and effectively. To address these challenges, this paper proposes FedCGS, a novel Federated learning algorithm that Capture Global feature Statistics leveraging pre-trained models. With global feature statistics, we achieve training-free and heterogeneity-resistant one-shot FL. Furthermore, we extend its application to personalization scenario, where clients only need execute one extra communication round with server to download global statistics. Extensive experimental results demonstrate the effectiveness of our methods across diverse data heterogeneity settings. Code is available at https://github.com/Yuqin-G/FedCGS.

Paper Structure

This paper contains 23 sections, 11 equations, 4 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. One-shot Federated Learning
  4. Federated Learning with pre-trained Models
  5. Method
  6. Problem Formulation
  7. Capture Global Feature Statistics
  8. FedCGS for global one-shot FL
  9. FedCGS for personalized one-shot FL
  10. Experiments
  11. Global one-shot FL
  12. Personalized one-shot FL
  13. In-deepth Study
  14. Discussion
  15. Communication Overhead
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Framework of our FedCGS. $S_i$ is the local statistics of client $i$ as shown in \ref{['sec:Capture Global Feature Statistics']}. We obtain global feature statistics through aggregating local feature statistics
  • Figure 2: Performance comparison using different classifier configurations.
  • Figure 3: Results of Feature expansion.
  • Figure 4: Results of inversion attacks on CIFAR100. Assume a client has only 4 "aquarium fish" samples, as shown in Ground Truth. The server attempts to reconstruct a specific data sample from this client. If the server has access to the Raw Feature of each sample, the reconstructed results are clear. However, when using our uploaded variables, the results are poor. The PSNR value (red) is displayed below each reconstructed image as a quantitative measure.