Revisiting Ensembling in One-Shot Federated Learning

TL;DR

This work introduces FENS, a novel federated ensembling scheme that approaches the accuracy of FL with the communication efficiency of OFL, and showcases the effectiveness of FENS through exhaustive experiments spanning several datasets and heterogeneity levels.

Abstract

Federated learning (FL) is an appealing approach to training machine learning models without sharing raw data. However, standard FL algorithms are iterative and thus induce a significant communication cost. One-shot federated learning (OFL) trades the iterative exchange of models between clients and the server with a single round of communication, thereby saving substantially on communication costs. Not surprisingly, OFL exhibits a performance gap in terms of accuracy with respect to FL, especially under high data heterogeneity. We introduce FENS, a novel federated ensembling scheme that approaches the accuracy of FL with the communication efficiency of OFL. Learning in FENS proceeds in two phases: first, clients train models locally and send them to the server, similar to OFL; second, clients collaboratively train a lightweight prediction aggregator model using FL. We showcase the effectiveness of FENS through exhaustive experiments spanning several datasets and heterogeneity levels. In the particular case of heterogeneously distributed CIFAR-10 dataset, FENS achieves up to a 26.9% higher accuracy over state-of-the-art (SOTA) OFL, being only 3.1% lower than FL. At the same time, FENS incurs at most 4.3x more communication than OFL, whereas FL is at least 10.9x more communication-intensive than FENS.

Figures (11)

• Figure 1: Fens in comparison to iterative and one-shot federated learning.
• Figure 2: Test accuracy and communication cost of OFL, Fens and FL on CIFAR-10 dataset under high data heterogeneity.
• Figure 3: Total communication cost of Fens against OFL baselines. The clients in Fens expend roughly $3.7 - 4.3 \times$ more than OFL in communication costs.
• Figure 4: Fens against iterative FL. The R indicates the number of global rounds, signifying the multi-round version of the OFL baseline. Fens achieves accuracy properties of iterative FL (FedAdam) with a modest increase in communication cost compared to OFL (FedKD). Numerical accuracy results are included in \ref{['tab:fens_vs_ifl']} (\ref{['appendix:numerical']}).
• Figure 5: Accuracy of Fens for increasing dataset size. Performance of Fens rapidly increases as the data volume increases. At high data heterogeneity, Fens matches iterative FL's accuracy.