The Gaussian-Head OFL Family: One-Shot Federated Learning from Client Global Statistics

TL;DR

GH-OFL tackles the inefficiency and privacy risks of multi-round federated learning by proposing a one-shot, server-centric framework that relies only on per-class sufficient statistics and a compressed random-projection sketch. The server constructs closed-form Gaussian discriminants (NB, LDA, QDA) from aggregated moments and augments them with data-free trained heads (FisherMix, Proto-Hyper) trained on synthetic Fisher-space samples. This approach is robust to non-IID partitions, reduces communication to a single round, and preserves privacy by avoiding raw data or client-side inference. Empirical results across CIFAR-10/100, CIFAR-100-C, SVHN, and NLP benchmarks demonstrate state-of-the-art robustness and accuracy in a strictly data-free setting, with scalable, privacy-friendly deployment potential in edge and IoT contexts.

Abstract

Classical Federated Learning relies on a multi-round iterative process of model exchange and aggregation between server and clients, with high communication costs and privacy risks from repeated model transmissions. In contrast, one-shot federated learning (OFL) alleviates these limitations by reducing communication to a single round, thereby lowering overhead and enhancing practical deployability. Nevertheless, most existing one-shot approaches remain either impractical or constrained, for example, they often depend on the availability of a public dataset, assume homogeneous client models, or require uploading additional data or model information. To overcome these issues, we introduce the Gaussian-Head OFL (GH-OFL) family, a suite of one-shot federated methods that assume class-conditional Gaussianity of pretrained embeddings. Clients transmit only sufficient statistics (per-class counts and first/second-order moments) and the server builds heads via three components: (i) Closed-form Gaussian heads (NB/LDA/QDA) computed directly from the received statistics; (ii) FisherMix, a linear head with cosine margin trained on synthetic samples drawn in an estimated Fisher subspace; and (iii) Proto-Hyper, a lightweight low-rank residual head that refines Gaussian logits via knowledge distillation on those synthetic samples. In our experiments, GH-OFL methods deliver state-of-the-art robustness and accuracy under strong non-IID skew while remaining strictly data-free.

Figures (6)

• Figure 1: Client-side flow with secure aggregation. Each device encodes images via a frozen ImageNet backbone, projects embeddings with a public RP ($z{=}xR$) and updates additively-aggregable stats in $z$: $N_c$, $A_c$, global $B$ and $S_c$/$D_c$ as required by the chosen heads. Secure aggregation reveals only $\sum_u(\cdot)$, which suffice to estimate means/covariances and run our Gaussian and Fisher-space heads without sharing raw data or gradients.
• Figure 2: Server-side pipelines for our GH-OFL family. Secure aggregation first collects class-wise sufficient statistics in the projected space $z$. Closed-form heads compute scores directly, while FisherMix and Proto-Hyper estimate a Fisher subspace, synthesize features and fit lightweight heads without any raw data. Shaded panels summarize per-method steps and outputs.
• Figure 3: Accuracy vs. overhead on CIFAR-10 (left) and CIFAR-100-C (right). For the overhead, we considered an estimate of upload bandwidth for client–server communication.
• Figure 4: Comparison of FedAvg (left) and FedProx (right) on CIFAR-10 under Dirichlet client partitions ($\alpha \in \{0.50,0.10,0.05\}$). FedProx exhibits smoother convergence under stronger non-IID heterogeneity while both methods require multiple rounds to stabilize.
• Figure 5: FedAvg (left) and FedProx (right) results on CIFAR-100 with Dirichlet client splits. Due to the fine-grained nature of CIFAR-100, accuracy grows more slowly and the gap between different $\alpha$ values is more pronounced. FedProx reduces oscillations caused by heterogeneous updates.