One-shot Federated Learning Methods: A Practical Guide

TL;DR

One-shot Federated Learning (OFL) restricts server-client communication to a single round, addressing privacy and bandwidth concerns while enabling collaborative training for large models such as LLMs; the paper formalizes the OFL objective with $ \min_{\bm{w}} F(\bm{w}) := \alpha_i \sum_{i \in [n]} F_i(\bm{w_i})$ and surveys how data heterogeneity and model heterogeneity challenge this setting. It introduces a novel taxonomy of OFL methods across Parameter Learning, Knowledge Distillation, Generative Models, Ensemble Methods, and Hybrid Methods, and analyzes the trade-offs and practical considerations of each category. The authors provide a thorough review of existing OFL techniques, discuss their convergence, privacy implications, and applicability to heterogeneous data and models, and outline future directions including data-free operation and scalability to large models. The work aims to guide researchers and practitioners in designing OFL systems suitable for real-world, distributed environments, including cloud-edge deployments and potential integration with LLMs.

Abstract

One-shot Federated Learning (OFL) is a distributed machine learning paradigm that constrains client-server communication to a single round, addressing privacy and communication overhead issues associated with multiple rounds of data exchange in traditional Federated Learning (FL). OFL demonstrates the practical potential for integration with future approaches that require collaborative training models, such as large language models (LLMs). However, current OFL methods face two major challenges: data heterogeneity and model heterogeneity, which result in subpar performance compared to conventional FL methods. Worse still, despite numerous studies addressing these limitations, a comprehensive summary is still lacking. To address these gaps, this paper presents a systematic analysis of the challenges faced by OFL and thoroughly reviews the current methods. We also offer an innovative categorization method and analyze the trade-offs of various techniques. Additionally, we discuss the most promising future directions and the technologies that should be integrated into the OFL field. This work aims to provide guidance and insights for future research.

Figures (1)

• Figure 1: A basic taxonomy of one-shot federated learning techniques. Note that Some hybrid methods employ multiple techniques.