Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

$r$Age-$k$: Communication-Efficient Federated Learning Using Age Factor

Matin Mortaheb, Priyanka Kaswan, Sennur Ulukus

TL;DR

A new communication-efficient algorithm that uses the age of information metric to simultaneously tackle both limitations of FL and surpassing other communication-efficient strategies in efficiency is introduced.

Abstract

Federated learning (FL) is a collaborative approach where multiple clients, coordinated by a parameter server (PS), train a unified machine-learning model. The approach, however, suffers from two key challenges: data heterogeneity and communication overhead. Data heterogeneity refers to inconsistencies in model training arising from heterogeneous data at different clients. Communication overhead arises from the large volumes of parameter updates exchanged between the PS and clients. Existing solutions typically address these challenges separately. This paper introduces a new communication-efficient algorithm that uses the age of information metric to simultaneously tackle both limitations of FL. We introduce age vectors at the PS, which keep track of how often the different model parameters are updated from the clients. The PS uses this to selectively request updates for specific gradient indices from each client. Further, the PS employs age vectors to identify clients with statistically similar data and group them into clusters. The PS combines the age vectors of the clustered clients to efficiently coordinate gradient index updates among clients within a cluster. We evaluate our approach using the MNIST and CIFAR10 datasets in highly non-i.i.d. settings. The experimental results show that our proposed method can expedite training, surpassing other communication-efficient strategies in efficiency.

$r$Age-$k$: Communication-Efficient Federated Learning Using Age Factor

TL;DR

A new communication-efficient algorithm that uses the age of information metric to simultaneously tackle both limitations of FL and surpassing other communication-efficient strategies in efficiency is introduced.

Abstract

Federated learning (FL) is a collaborative approach where multiple clients, coordinated by a parameter server (PS), train a unified machine-learning model. The approach, however, suffers from two key challenges: data heterogeneity and communication overhead. Data heterogeneity refers to inconsistencies in model training arising from heterogeneous data at different clients. Communication overhead arises from the large volumes of parameter updates exchanged between the PS and clients. Existing solutions typically address these challenges separately. This paper introduces a new communication-efficient algorithm that uses the age of information metric to simultaneously tackle both limitations of FL. We introduce age vectors at the PS, which keep track of how often the different model parameters are updated from the clients. The PS uses this to selectively request updates for specific gradient indices from each client. Further, the PS employs age vectors to identify clients with statistically similar data and group them into clusters. The PS combines the age vectors of the clustered clients to efficiently coordinate gradient index updates among clients within a cluster. We evaluate our approach using the MNIST and CIFAR10 datasets in highly non-i.i.d. settings. The experimental results show that our proposed method can expedite training, surpassing other communication-efficient strategies in efficiency.

Paper Structure

This paper contains 8 sections, 4 equations, 5 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. System Model and Problem Formulation
  3. Convergence of rAge-k
  4. Experimental Results
  5. Dataset Specifications
  6. Model and Hyperparameters
  7. System Performance
  8. Conclusion and Discussion

Figures (5)

  • Figure 1: System model for rAge-k framework.
  • Figure 2: Heatmap of the connectivity matrix achieved by the DBSCAN method over different epochs, (a) iteration 1, (b) iteration 21, (c) iteration 41, (d) iteration 61.
  • Figure 3: (a) Accuracy (in percentage) and (b) loss (averaged over all 10 users) over training iterations with MNIST dataset.
  • Figure 4: Heatmap of the connectivity matrix achieved by the DBSCAN method over different epochs, (a) iteration 1, (b) iteration 201
  • Figure 5: (a) Accuracy (in percentage) and (b) loss (averaged over all 6 users) over training iterations with CIFAR10 dataset.