Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedNMUT -- Federated Noisy Model Update Tracking Convergence Analysis

Vishnu Pandi Chellapandi, Antesh Upadhyay, Abolfazl Hashemi, Stanislaw H. Żak

TL;DR

FedNMUT proves the capability of the proposed algorithm to counteract the negative effects of communication noise in a decentralized learning framework and is superior to the existing state-of-the-art methods and conventional parameter-mixing approaches in dealing with imperfect information sharing.

Abstract

A novel Decentralized Noisy Model Update Tracking Federated Learning algorithm (FedNMUT) is proposed that is tailored to function efficiently in the presence of noisy communication channels that reflect imperfect information exchange. This algorithm uses gradient tracking to minimize the impact of data heterogeneity while minimizing communication overhead. The proposed algorithm incorporates noise into its parameters to mimic the conditions of noisy communication channels, thereby enabling consensus among clients through a communication graph topology in such challenging environments. FedNMUT prioritizes parameter sharing and noise incorporation to increase the resilience of decentralized learning systems against noisy communications. Theoretical results for the smooth non-convex objective function are provided by us, and it is shown that the $ε-$stationary solution is achieved by our algorithm at the rate of $\mathcal{O}\left(\frac{1}{\sqrt{T}}\right)$, where $T$ is the total number of communication rounds. Additionally, via empirical validation, we demonstrated that the performance of FedNMUT is superior to the existing state-of-the-art methods and conventional parameter-mixing approaches in dealing with imperfect information sharing. This proves the capability of the proposed algorithm to counteract the negative effects of communication noise in a decentralized learning framework.

FedNMUT -- Federated Noisy Model Update Tracking Convergence Analysis

TL;DR

FedNMUT proves the capability of the proposed algorithm to counteract the negative effects of communication noise in a decentralized learning framework and is superior to the existing state-of-the-art methods and conventional parameter-mixing approaches in dealing with imperfect information sharing.

Abstract

A novel Decentralized Noisy Model Update Tracking Federated Learning algorithm (FedNMUT) is proposed that is tailored to function efficiently in the presence of noisy communication channels that reflect imperfect information exchange. This algorithm uses gradient tracking to minimize the impact of data heterogeneity while minimizing communication overhead. The proposed algorithm incorporates noise into its parameters to mimic the conditions of noisy communication channels, thereby enabling consensus among clients through a communication graph topology in such challenging environments. FedNMUT prioritizes parameter sharing and noise incorporation to increase the resilience of decentralized learning systems against noisy communications. Theoretical results for the smooth non-convex objective function are provided by us, and it is shown that the stationary solution is achieved by our algorithm at the rate of , where is the total number of communication rounds. Additionally, via empirical validation, we demonstrated that the performance of FedNMUT is superior to the existing state-of-the-art methods and conventional parameter-mixing approaches in dealing with imperfect information sharing. This proves the capability of the proposed algorithm to counteract the negative effects of communication noise in a decentralized learning framework.
Paper Structure (15 sections, 11 theorems, 72 equations, 6 figures, 2 algorithms)

This paper contains 15 sections, 11 theorems, 72 equations, 6 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Motivation
  3. Literature Overview
  4. Paper's Contributions
  5. Problem Statement, Algorithm, and Assumptions
  6. Problem Statement
  7. Assumptions
  8. Convergence Analysis
  9. Numerical Experiments
  10. Summary and Conclusions
  11. Assumptions, Definitions, and Background Results
  12. Assumptions
  13. Technical Results
  14. Main Results
  15. Proof of Corollary

Key Result

lemma 1

Suppose Assumption as3 holds and let $\Bar{b}^t = B^t \frac{1}{n} \mathds{1}$, where $\mathds{1}$ is a vector of all ones, then for all $t$, we have

Figures (6)

  • Figure 1: Loss versus iterations for various $\mu$ values for different communication topologies (No noise scenario).
  • Figure 2: Consensus error versus iterationsfor various $\mu$ values for different communication topologies (No noise scenario).
  • Figure 3: Loss versus iterations with and without noise (Var=0.005) for different communication topologies. $\mu$ = 0.02
  • Figure 4: Consensus error versus iterations with and without noise (Var=0.005) for different communication topologies. $\mu$ = 0.02. Note the different Y-axis scale in Fig (a) as compared with Fig (b) and (c) for better readability.
  • Figure 5: Loss versus iterations with and without noise (Var=0.01) for different communication topologies. $\mu$ = 0.02
  • ...and 1 more figures

Theorems & Definitions (18)

  • Definition 1: Mixing matrix
  • lemma 1
  • lemma 2
  • lemma 3
  • lemma 4
  • Theorem 1: Smooth non-convex cases for FedNMUT
  • proof
  • Corollary 1
  • proof
  • lemma 1
  • ...and 8 more