Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Communication-Efficient Model Aggregation with Layer Divergence Feedback in Federated Learning

Liwei Wang, Jun Li, Wen Chen, Qingqing Wu, Ming Ding

TL;DR

FedLDF tackles the high communication cost of federated learning by employing layer-wise divergence feedback to selectively upload model layers. The method computes layer divergences $DeltaTheta_{k,l}^{t+1} = |Theta_{k,l}^{t+1} - Theta_hat_l^t|$ and uses a top-$n$ strategy to decide which layers to transmit, followed by layer-wise aggregation weighted by client data sizes. A convergence bound expresses how the gap between FedLDF and FedAvg depends on the participation ratio $n/K$ and other problem constants, with the bound shrinking as more layers participate, ultimately recovering FedAvg when $n=K$. Empirical results on CIFAR-10 with a VGG-9 backbone show up to 80% communication savings while maintaining or improving accuracy across IID and Non-IID data, highlighting FedLDF's practical potential for communication-constrained FL deployments.

Abstract

Federated Learning (FL) facilitates collaborative machine learning by training models on local datasets, and subsequently aggregating these local models at a central server. However, the frequent exchange of model parameters between clients and the central server can result in significant communication overhead during the FL training process. To solve this problem, this paper proposes a novel FL framework, the Model Aggregation with Layer Divergence Feedback mechanism (FedLDF). Specifically, we calculate model divergence between the local model and the global model from the previous round. Then through model layer divergence feedback, the distinct layers of each client are uploaded and the amount of data transferred is reduced effectively. Moreover, the convergence bound reveals that the access ratio of clients has a positive correlation with model performance. Simulation results show that our algorithm uploads local models with reduced communication overhead while upholding a superior global model performance.

Communication-Efficient Model Aggregation with Layer Divergence Feedback in Federated Learning

TL;DR

FedLDF tackles the high communication cost of federated learning by employing layer-wise divergence feedback to selectively upload model layers. The method computes layer divergences and uses a top- strategy to decide which layers to transmit, followed by layer-wise aggregation weighted by client data sizes. A convergence bound expresses how the gap between FedLDF and FedAvg depends on the participation ratio and other problem constants, with the bound shrinking as more layers participate, ultimately recovering FedAvg when . Empirical results on CIFAR-10 with a VGG-9 backbone show up to 80% communication savings while maintaining or improving accuracy across IID and Non-IID data, highlighting FedLDF's practical potential for communication-constrained FL deployments.

Abstract

Federated Learning (FL) facilitates collaborative machine learning by training models on local datasets, and subsequently aggregating these local models at a central server. However, the frequent exchange of model parameters between clients and the central server can result in significant communication overhead during the FL training process. To solve this problem, this paper proposes a novel FL framework, the Model Aggregation with Layer Divergence Feedback mechanism (FedLDF). Specifically, we calculate model divergence between the local model and the global model from the previous round. Then through model layer divergence feedback, the distinct layers of each client are uploaded and the amount of data transferred is reduced effectively. Moreover, the convergence bound reveals that the access ratio of clients has a positive correlation with model performance. Simulation results show that our algorithm uploads local models with reduced communication overhead while upholding a superior global model performance.
Paper Structure (8 sections, 2 theorems, 19 equations, 4 figures, 1 algorithm)

This paper contains 8 sections, 2 theorems, 19 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Framework and convergence analysis
  3. FedLDF Algorithm
  4. Convergence Analysis
  5. Experiment Result
  6. Experiment Settings
  7. Performance Analysis
  8. Conclusion

Key Result

Theorem 1

Given the number of clients $K$ participated in local training, the number of clients $n$ that upload the $l$-th layer and the learning rate $\eta$, with the assumptions, when the condition $0 < \xi_2 < \frac{1}{2(1+\beta)\eta^2 L^2}$ is satisfied, then the convergence upper bound of Algorithm 1 is where A = $2\xi_2\eta^2 L^2(1-\frac{n}{K})\left[1+\beta(1-\frac{n}{K}) \right]$ and B = $\frac{\xi_

Figures (4)

  • Figure 1: FedLDF framework.
  • Figure 2: Under the five-layer neural network of five clients, each client uploads the different local model layers in the aggregation$(n=3,K=5)$.
  • Figure 3: Test error comparisons base on CIFAR10, IID.
  • Figure 4: Test error comparisons base on CIFAR10, Non-IID.

Theorems & Definitions (4)

  • Theorem 1
  • proof : Proof of Theorem \ref{['theorem1']}
  • Lemma 1
  • proof : Proof of Lemma \ref{['lemma1']}