Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fed-ZOE: Communication-Efficient Over-the-Air Federated Learning via Zeroth-Order Estimation

Jonggyu Jang, Hyeonsu Lyu, David J. Love, Hyun Jong Yang

TL;DR

Fed-ZOE tackles the high communication cost of over-the-air federated learning for large neural networks by compressing local model updates through randomized gradient estimation. Edge devices send $L$ scalar projections instead of full gradient vectors, and the PS reconstructs the aggregate gradient via the RGE framework while preserving the superposition property. The authors provide convergence guarantees showing an $\mathcal{O}(1/\sqrt{TI})$ rate under realistic assumptions and demonstrate strong empirical performance across multiple datasets with drastically reduced uplink/downlink overhead. The approach achieves near-Fed-OtA accuracy with a compression ratio as low as $L/S \approx 0.07\%$, highlighting significant practical impact for scalable OtA-FL in 6G+ networks.

Abstract

As 6G and beyond networks grow increasingly complex and interconnected, federated learning (FL) emerges as an indispensable paradigm for securely and efficiently leveraging decentralized edge data for AI. By virtue of the superposition property of communication signals, over-the-air FL (OtA-FL) achieves constant communication overhead irrespective of the number of edge devices (EDs). However, training neural networks over the air still incurs substantial communication costs, as the number of transmitted symbols equals the number of trainable parameters. To alleviate this issue, the most straightforward approach is to reduce the number of transmitted symbols by 1) gradient compression and 2) gradient sparsification. Unfortunately, these methods are incompatible with OtA-FL due to the loss of its superposition property. In this work, we introduce federated zeroth-order estimation (Fed-ZOE), an efficient framework inspired by the randomized gradient estimator (RGE) commonly used in zeroth-order optimization (ZOO). In FedZOE, EDs perform local weight updates as in standard FL, but instead of transmitting full gradient vectors, they send compressed local model update vectors in the form of several scalar-valued inner products between the local model update vectors and random vectors. These scalar values enable the parameter server (PS) to reconstruct the gradient using the RGE trick with highly reduced overhead, as well as preserving the superposition property. Unlike conventional ZOO leveraging RGE for step-wise gradient descent, Fed-ZOE compresses local model update vectors before transmission, thereby achieving higher accuracy and computational efficiency. Numerical evaluations using ResNet-18 on datasets such as CIFAR-10, TinyImageNet, SVHN, CIFAR-100, and Brain-CT demonstrate that Fed-ZOE achieves performance comparable to Fed-OtA while drastically reducing communication costs.

Fed-ZOE: Communication-Efficient Over-the-Air Federated Learning via Zeroth-Order Estimation

TL;DR

Fed-ZOE tackles the high communication cost of over-the-air federated learning for large neural networks by compressing local model updates through randomized gradient estimation. Edge devices send scalar projections instead of full gradient vectors, and the PS reconstructs the aggregate gradient via the RGE framework while preserving the superposition property. The authors provide convergence guarantees showing an rate under realistic assumptions and demonstrate strong empirical performance across multiple datasets with drastically reduced uplink/downlink overhead. The approach achieves near-Fed-OtA accuracy with a compression ratio as low as , highlighting significant practical impact for scalable OtA-FL in 6G+ networks.

Abstract

As 6G and beyond networks grow increasingly complex and interconnected, federated learning (FL) emerges as an indispensable paradigm for securely and efficiently leveraging decentralized edge data for AI. By virtue of the superposition property of communication signals, over-the-air FL (OtA-FL) achieves constant communication overhead irrespective of the number of edge devices (EDs). However, training neural networks over the air still incurs substantial communication costs, as the number of transmitted symbols equals the number of trainable parameters. To alleviate this issue, the most straightforward approach is to reduce the number of transmitted symbols by 1) gradient compression and 2) gradient sparsification. Unfortunately, these methods are incompatible with OtA-FL due to the loss of its superposition property. In this work, we introduce federated zeroth-order estimation (Fed-ZOE), an efficient framework inspired by the randomized gradient estimator (RGE) commonly used in zeroth-order optimization (ZOO). In FedZOE, EDs perform local weight updates as in standard FL, but instead of transmitting full gradient vectors, they send compressed local model update vectors in the form of several scalar-valued inner products between the local model update vectors and random vectors. These scalar values enable the parameter server (PS) to reconstruct the gradient using the RGE trick with highly reduced overhead, as well as preserving the superposition property. Unlike conventional ZOO leveraging RGE for step-wise gradient descent, Fed-ZOE compresses local model update vectors before transmission, thereby achieving higher accuracy and computational efficiency. Numerical evaluations using ResNet-18 on datasets such as CIFAR-10, TinyImageNet, SVHN, CIFAR-100, and Brain-CT demonstrate that Fed-ZOE achieves performance comparable to Fed-OtA while drastically reducing communication costs.

Paper Structure

This paper contains 16 sections, 3 theorems, 54 equations, 9 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. System Model
  3. Proposed Method
  4. Local Update
  5. Local Model Update Vector Compression
  6. Global Aggregation
  7. Downlink Broadcast
  8. Theoretical Results
  9. Numerical Results
  10. Experimental Setup
  11. Experimental Results
  12. Datasets Other Than CIFAR-10
  13. Conclusion
  14. Beamforming Optimization
  15. Proof of Proposition \ref{['prop:proposition_1']}
  16. ...and 1 more sections

Key Result

Lemma 1

From the assumptions (A3) and (A4) in Assumption assumption:1, we have

Figures (9)

  • Figure 1: Overview of the proposed gradient compression and reconstruction framework in Fed-ZOE. The left panel illustrates the gradient compression process at each ED, where local model update vectors are projected onto randomized vectors, producing a series of scalar values of transmission. The central penal depicts the aggregation of these scalar values across all EDs. The right panel shows the gradient reconstruction process at the PS via RGE.
  • Figure 2: Convergence evaluation of the proposed method and baseline methods. For this experiment, we train a ResNet-18 model with the CIFAR-10 dataset. In \ref{['fig:convergence_cifar10_a']}, the test accuracy for the communication rounds is depicted. On the other hand, in \ref{['fig:convergence_cifar10_b']}, the test accuracy for the total communication and computation loads is depicted.
  • Figure 3: Test accuracy of the trained ResNet-18 models with various values of learning rate $\alpha$ and the number of RGE samples $L$. In this experiment, the proposed method is only implemented with $T=200$.
  • Figure 4: Communication and computation loads of the FL methods. The CIFAR-10 dataset and ResNet-18 model are used in this experiment. The unit is seconds.
  • Figure 5: Convergence of the proposed method for various numbers of the RGE samples $L\in\{512, 1024, 2048, 4096, 8192, 16384\}$.
  • ...and 4 more figures

Theorems & Definitions (4)

  • Definition 1: Randomized gradient estimation nesterov2017random
  • Lemma 1
  • Proposition 1
  • Theorem 1: Convergence