Enhancing Federated Learning Convergence with Dynamic Data Queue and Data Entropy-driven Participant Selection

TL;DR

This research presents a method to improve convergence in FL by creating a global subset of data on the server and dynamically distributing it across devices using a dynamic data queue-driven FL (DDFL), and provides a convergence analysis of this approach to justify their viability in practical FL scenarios.

Abstract

Federated Learning (FL) is a decentralized approach for collaborative model training on edge devices. This distributed method of model training offers advantages in privacy, security, regulatory compliance, and cost-efficiency. Our emphasis in this research lies in addressing statistical complexity in FL, especially when the data stored locally across devices is not identically and independently distributed (non-IID). We have observed an accuracy reduction of up to approximately 10\% to 30\%, particularly in skewed scenarios where each edge device trains with only 1 class of data. This reduction is attributed to weight divergence, quantified using the Euclidean distance between device-level class distributions and the population distribution, resulting in a bias term (\(δ_k\)). As a solution, we present a method to improve convergence in FL by creating a global subset of data on the server and dynamically distributing it across devices using a Dynamic Data queue-driven Federated Learning (DDFL). Next, we leverage Data Entropy metrics to observe the process during each training round and enable reasonable device selection for aggregation. Furthermore, we provide a convergence analysis of our proposed DDFL to justify their viability in practical FL scenarios, aiming for better device selection, a non-sub-optimal global model, and faster convergence. We observe that our approach results in a substantial accuracy boost of approximately 5\% for the MNIST dataset, around 18\% for CIFAR-10, and 20\% for CIFAR-100 with a 10\% global subset of data, outperforming the state-of-the-art (SOTA) aggregation algorithms.

Figures (6)

• Figure 1: Architectural overview of proposed FL for IoT network
• Figure 2: Proposed Federated Learning Framework (DDFL)
• Figure 3: Fig. (a) shows the initial Data Entropy in IID and non-IID data Distributions. Fig. (b) shows the data Entropy distribution for devices in each training round in the proposed DDFL. The colour bar is introduced to the respective epoch.
• Figure 4: Figures show the Test accuracy over the epoch of FedAvg with IID and non-IID setting, DDFL, and the warmup model with the non-IID setting for MNIST, CIFAR-10 and CIFAR-100 datasets.
• Figure 5: Comparison of accuracies of DDFL approach against MNIST, CIFAR-10 and CIFAR-100 datasets under different batch sizes