Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedCore: Straggler-Free Federated Learning with Distributed Coresets

Hongpeng Guo, Haotian Gu, Xiaoyang Wang, Bo Chen, Eun Kyung Lee, Tamar Eilam, Deming Chen, Klara Nahrstedt

TL;DR

FedCore addresses the straggler inefficiency in federated learning by constructing distributed coresets on each client to reduce the data processed per round while preserving privacy. It reframes coreset optimization as a k-medoids problem in gradient space and accelerates coreset generation with gradient-approximation techniques, enabling low-overhead adaptation as models evolve. The authors provide convergence guarantees that quantify the trade-off between coreset approximation error and federated optimization error, and demonstrate up to 8x training-time reductions without accuracy loss across MNIST, Shakespeare, and synthetic benchmarks. Overall, FedCore offers a privacy-preserving, scalable solution that integrates with existing FL frameworks and advances straggler-resilient learning through distributed coresets.

Abstract

Federated learning (FL) is a machine learning paradigm that allows multiple clients to collaboratively train a shared model while keeping their data on-premise. However, the straggler issue, due to slow clients, often hinders the efficiency and scalability of FL. This paper presents FedCore, an algorithm that innovatively tackles the straggler problem via the decentralized selection of coresets, representative subsets of a dataset. Contrary to existing centralized coreset methods, FedCore creates coresets directly on each client in a distributed manner, ensuring privacy preservation in FL. FedCore translates the coreset optimization problem into a more tractable k-medoids clustering problem and operates distributedly on each client. Theoretical analysis confirms FedCore's convergence, and practical evaluations demonstrate an 8x reduction in FL training time, without compromising model accuracy. Our extensive evaluations also show that FedCore generalizes well to existing FL frameworks.

FedCore: Straggler-Free Federated Learning with Distributed Coresets

TL;DR

FedCore addresses the straggler inefficiency in federated learning by constructing distributed coresets on each client to reduce the data processed per round while preserving privacy. It reframes coreset optimization as a k-medoids problem in gradient space and accelerates coreset generation with gradient-approximation techniques, enabling low-overhead adaptation as models evolve. The authors provide convergence guarantees that quantify the trade-off between coreset approximation error and federated optimization error, and demonstrate up to 8x training-time reductions without accuracy loss across MNIST, Shakespeare, and synthetic benchmarks. Overall, FedCore offers a privacy-preserving, scalable solution that integrates with existing FL frameworks and advances straggler-resilient learning through distributed coresets.

Abstract

Federated learning (FL) is a machine learning paradigm that allows multiple clients to collaboratively train a shared model while keeping their data on-premise. However, the straggler issue, due to slow clients, often hinders the efficiency and scalability of FL. This paper presents FedCore, an algorithm that innovatively tackles the straggler problem via the decentralized selection of coresets, representative subsets of a dataset. Contrary to existing centralized coreset methods, FedCore creates coresets directly on each client in a distributed manner, ensuring privacy preservation in FL. FedCore translates the coreset optimization problem into a more tractable k-medoids clustering problem and operates distributedly on each client. Theoretical analysis confirms FedCore's convergence, and practical evaluations demonstrate an 8x reduction in FL training time, without compromising model accuracy. Our extensive evaluations also show that FedCore generalizes well to existing FL frameworks.
Paper Structure (38 sections, 5 theorems, 33 equations, 8 figures, 2 tables, 1 algorithm)

This paper contains 38 sections, 5 theorems, 33 equations, 8 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivations.
  3. Methods and Results.
  4. Related Works
  5. Coreset Methods for Deep Learning.
  6. Straggler Prevention in Federated Learning.
  7. Preliminaries
  8. Federated Leaning System Setup
  9. Distributed Coresets for Federated Learning.
  10. FedCore Algorithm and System
  11. FedCore Algorithm Overview.
  12. Upper Bounding Dissimilarity Estimation with K-Medoids
  13. Accelerating Coreset Generation with Gradient Approximation.
  14. Convex Machine Learning Models.
  15. Deep Neural Networks.
  16. ...and 23 more sections

Key Result

Theorem 5.1

Assume that for any $i\in U$, the loss function ${\cal L}^i$ is $L$-smooth and $\mu$-strongly convex, and the coreset $\left(S^{i,*}, \delta^{i,*}\right)$ constructed in FedCore is an $\epsilon$-approximation to the full-set, i.e. Consider FedCore with $R$ rounds with each round containing $E$ epochs. Set the learning rate $\eta_t=\Omega(1/t)$ for $t\in\{1,2,\cdots,ER\}$. The model $w_\text{out}$

Figures (8)

  • Figure 1: An example workflow of FedCore encompasses a single training round consisting of 6 epochs.
  • Figure 2: Statistics of the benchmarks
  • Figure 3: Distribution of training samples per client
  • Figure 4: The training loss curves for FedAvg-DS, FedCore, and FedProx at 10% and 30% stragglers.
  • Figure 5: Round length distribution on MNIST benchmark, 30% stragglers. The y-axis is presented in log-scale for better illustration.
  • ...and 3 more figures

Theorems & Definitions (5)

  • Theorem 5.1
  • Theorem A.7
  • Lemma A.8
  • Lemma A.9
  • Lemma A.10