Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Communication and Energy Efficient Federated Learning using Zero-Order Optimization Technique

Elissa Mhanna, Mohamad Assaad

TL;DR

This paper proposes a zero-order (ZO) optimization method that requires the upload of a quantized single scalar per iteration by each device instead of the whole gradient vector, and proves its theoretical convergence and finds an upper bound on its convergence rate in the non-convex setting.

Abstract

Federated learning (FL) is a popular machine learning technique that enables multiple users to collaboratively train a model while maintaining the user data privacy. A significant challenge in FL is the communication bottleneck in the upload direction, and thus the corresponding energy consumption of the devices, attributed to the increasing size of the model/gradient. In this paper, we address this issue by proposing a zero-order (ZO) optimization method that requires the upload of a quantized single scalar per iteration by each device instead of the whole gradient vector. We prove its theoretical convergence and find an upper bound on its convergence rate in the non-convex setting, and we discuss its implementation in practical scenarios. Our FL method and the corresponding convergence analysis take into account the impact of quantization and packet dropping due to wireless errors. We show also the superiority of our method, in terms of communication overhead and energy consumption, as compared to standard gradient-based FL methods.

Communication and Energy Efficient Federated Learning using Zero-Order Optimization Technique

TL;DR

This paper proposes a zero-order (ZO) optimization method that requires the upload of a quantized single scalar per iteration by each device instead of the whole gradient vector, and proves its theoretical convergence and finds an upper bound on its convergence rate in the non-convex setting.

Abstract

Federated learning (FL) is a popular machine learning technique that enables multiple users to collaboratively train a model while maintaining the user data privacy. A significant challenge in FL is the communication bottleneck in the upload direction, and thus the corresponding energy consumption of the devices, attributed to the increasing size of the model/gradient. In this paper, we address this issue by proposing a zero-order (ZO) optimization method that requires the upload of a quantized single scalar per iteration by each device instead of the whole gradient vector. We prove its theoretical convergence and find an upper bound on its convergence rate in the non-convex setting, and we discuss its implementation in practical scenarios. Our FL method and the corresponding convergence analysis take into account the impact of quantization and packet dropping due to wireless errors. We show also the superiority of our method, in terms of communication overhead and energy consumption, as compared to standard gradient-based FL methods.
Paper Structure (19 sections, 7 theorems, 42 equations, 2 figures, 1 algorithm)

This paper contains 19 sections, 7 theorems, 42 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. System Model and proposed algorithm
  3. System Model
  4. Algorithm
  5. Energy and communication overhead reduction
  6. Communication overhead
  7. Convergence time
  8. Energy Consumption
  9. Convergence analysis
  10. Numerical Results
  11. Conclusion
  12. Analysis of $\mathbb{E}(g_k)$
  13. Proof of Lemma \ref{['lemma-erasure']}: Effect of the Packet Erasure Channel
  14. Proof of Lemma \ref{['biased_estimators']}: Biased Estimator
  15. Proof of Lemma \ref{['norm']}: Expected Norm Squared of the Estimated Gradient
  16. ...and 4 more sections

Key Result

Lemma 4.6

By Assumption objective_fct, we know that the objective function $\theta\longmapsto F(\theta)$ is $L$-smooth for some positive constant $L$, $\|\nabla F(\theta)-\nabla F(\theta')\|\leq L\|\theta-\theta'\|, \;\forall \theta,\theta'\in\mathbb{R}^d,$ or equivalently, $F(\theta)\leq F(\theta')+\langle\n

Figures (2)

  • Figure 1: Total Energy Consumption (joule)
  • Figure 2: Total Convergence Time (sec)

Theorems & Definitions (10)

  • Remark 2.1
  • Example 4.4
  • Lemma 4.6
  • Example 4.8
  • Theorem 4.9
  • Theorem 4.10
  • Lemma A.1
  • Lemma A.2
  • Lemma A.3
  • Lemma A.4