Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SA-PEF: Step-Ahead Partial Error Feedback for Efficient Federated Learning

Dawit Kiros Redie, Reza Arablouei, Stefan Werner

TL;DR

SA-PEF introduces Step-Ahead Partial Error Feedback for efficient federated learning by blending a step-ahead residual preview with partial residual carry‑over. The method smoothly interpolates between EF and SAEF with a tunable coefficient $\alpha_r$, yielding faster early convergence while preserving EF’s long-term stability under biased $(\\delta)$-contractive compression and local updates. Theoretical guarantees establish nonconvex stationarity with a convergence rate of $O\bigl(1/(\\eta\\eta_0 T R)\bigr)$ and a residual contraction $\\rho_r$ that improves over EF; practical guidance centers on choosing $\\alpha_r$ near its optimum given the current stepsize. Empirically, SA-PEF consistently outperforms EF and SAEF in terms of accuracy per communication budget across CIFAR-10/100 and Tiny-ImageNet, across varying heterogeneity and participation, while retaining low overhead. This makes SA-PEF a practical and robust drop-in enhancement for compressed FL, particularly in regimes with aggressive compression and data non-IIDness.

Abstract

Biased gradient compression with error feedback (EF) reduces communication in federated learning (FL), but under non-IID data, the residual error can decay slowly, causing gradient mismatch and stalled progress in the early rounds. We propose step-ahead partial error feedback (SA-PEF), which integrates step-ahead (SA) correction with partial error feedback (PEF). SA-PEF recovers EF when the step-ahead coefficient $α=0$ and step-ahead EF (SAEF) when $α=1$. For non-convex objectives and $δ$-contractive compressors, we establish a second-moment bound and a residual recursion that guarantee convergence to stationarity under heterogeneous data and partial client participation. The resulting rates match standard non-convex Fed-SGD guarantees up to constant factors, achieving $O((η,η_0TR)^{-1})$ convergence to a variance/heterogeneity floor with a fixed inner step size. Our analysis reveals a step-ahead-controlled residual contraction $ρ_r$ that explains the observed acceleration in the early training phase. To balance SAEF's rapid warm-up with EF's long-term stability, we select $α$ near its theory-predicted optimum. Experiments across diverse architectures and datasets show that SA-PEF consistently reaches target accuracy faster than EF.

SA-PEF: Step-Ahead Partial Error Feedback for Efficient Federated Learning

TL;DR

SA-PEF introduces Step-Ahead Partial Error Feedback for efficient federated learning by blending a step-ahead residual preview with partial residual carry‑over. The method smoothly interpolates between EF and SAEF with a tunable coefficient , yielding faster early convergence while preserving EF’s long-term stability under biased -contractive compression and local updates. Theoretical guarantees establish nonconvex stationarity with a convergence rate of and a residual contraction that improves over EF; practical guidance centers on choosing near its optimum given the current stepsize. Empirically, SA-PEF consistently outperforms EF and SAEF in terms of accuracy per communication budget across CIFAR-10/100 and Tiny-ImageNet, across varying heterogeneity and participation, while retaining low overhead. This makes SA-PEF a practical and robust drop-in enhancement for compressed FL, particularly in regimes with aggressive compression and data non-IIDness.

Abstract

Biased gradient compression with error feedback (EF) reduces communication in federated learning (FL), but under non-IID data, the residual error can decay slowly, causing gradient mismatch and stalled progress in the early rounds. We propose step-ahead partial error feedback (SA-PEF), which integrates step-ahead (SA) correction with partial error feedback (PEF). SA-PEF recovers EF when the step-ahead coefficient and step-ahead EF (SAEF) when . For non-convex objectives and -contractive compressors, we establish a second-moment bound and a residual recursion that guarantee convergence to stationarity under heterogeneous data and partial client participation. The resulting rates match standard non-convex Fed-SGD guarantees up to constant factors, achieving convergence to a variance/heterogeneity floor with a fixed inner step size. Our analysis reveals a step-ahead-controlled residual contraction that explains the observed acceleration in the early training phase. To balance SAEF's rapid warm-up with EF's long-term stability, we select near its theory-predicted optimum. Experiments across diverse architectures and datasets show that SA-PEF consistently reaches target accuracy faster than EF.
Paper Structure (52 sections, 11 theorems, 102 equations, 15 figures, 3 tables, 1 algorithm)

This paper contains 52 sections, 11 theorems, 102 equations, 15 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Error feedback and compressed optimization.
  4. Local updates in FL.
  5. Compression with local updates.
  6. Step-ahead error feedback.
  7. Proposed algorithm
  8. Algorithm overview.
  9. Relation to EF and SAEF.
  10. Residual contraction.
  11. Convergence Analysis
  12. Discussion.
  13. Comparison to EF under PP.
  14. Why step-ahead helps (and how much).
  15. Experiments
  16. ...and 37 more sections

Key Result

Theorem 1

Assume A1--A3 and let the compressor satisfy Definition def:compression-operator with parameter $\delta\ge1$. Run SA-PEF for $R\ge1$ rounds with a constant inner-loop stepsize $\eta_r\equiv\eta_0$, and set $s_0:=\eta_0LT\le \tfrac{1}{8}$. Suppose further that $18\,\beta^{2}s_0^{2}\le \tfrac{1}{8}$ a and the effective error constant where $\mathcal{E}_{\max}:=\sup_{r}\mathcal{E}_r$ is the maximum

Figures (15)

  • Figure 1: Test accuracy vs number of rounds (row 1) and communicated GB (row 2) on the CIFAR-10 dataset using ResNet-9.
  • Figure 2: Test accuracy vs number of rounds (row 1) and communicated GB (row 2) on the CIFAR-100 dataset using ResNet-18 and $\gamma{=}0.1$.
  • Figure 3: Sensitivity analysis of step-ahead coefficient $\alpha$.
  • Figure 4: Gradient mismatch for different algorithms.
  • Figure 5: Test accuracy vs number of rounds (row 1) and communicated GB (row 2) on the CIFAR-10 dataset using ResNet-9 and $\gamma{=}0.1$.
  • ...and 10 more figures

Theorems & Definitions (20)

  • Definition 1: Compression Operator
  • Theorem 1: Stationary-point bound with constant inner-loop step
  • Remark 1: Partial participation (PP)
  • proof
  • Lemma 1: Sufficient Conditions for Descent
  • proof
  • Lemma 2: Local-model drift under SA-PEF
  • proof
  • Lemma 3: Second moment of the shifted average update
  • proof
  • ...and 10 more