FedECADO: A Dynamical System Model of Federated Learning

TL;DR

This work proposes FedECADO, a new algorithm inspired by a dynamical system representation of the federated learning process that addresses non-IID data distribution through an aggregate sensitivity model that reflects the amount of data processed by each client.

Abstract

Federated learning harnesses the power of distributed optimization to train a unified machine learning model across separate clients. However, heterogeneous data distributions and computational workloads can lead to inconsistent updates and limit model performance. This work tackles these challenges by proposing FedECADO, a new algorithm inspired by a dynamical system representation of the federated learning process. FedECADO addresses non-IID data distribution through an aggregate sensitivity model that reflects the amount of data processed by each client. To tackle heterogeneous computing, we design a multi-rate integration method with adaptive step-size selections that synchronizes active client updates in continuous time. Compared to prominent techniques, including FedProx and FedNova, FedECADO achieves higher classification accuracies in numerous heterogeneous scenarios.

Figures (6)

• Figure 1: ECADO models federated learning as an equivalent circuit, where node voltages represent state variables, $\mathbf{x}_i$, and gradients, $\nabla f_i(\mathbf{x}_i)$, are voltage-controlled current sources. Using circuit insights, the gradient flow equations \ref{['eq:gd_flow1']},\ref{['eq:gd_flow']} are modified by introducing an inductor (with an inductance of $L$) between the central agent state, $\mathbf{x}_c$, and the state of each sub-problem, $\mathbf{x}_i$. The resulting gradient flow equations \ref{['eq:central_agent_ode']},\ref{['eq:client_ode']},\ref{['eq:inductor_ode']} are mapped to the equivalent circuit shown.
• Figure 2: Heterogeneous computation among three clients leads to simulation for different time windows ($T_1,T_2,T_3$). The final states ($x_1(T_1),x_2(T_2),x_3(T_3)$) are communicated to the central agent, resulting in asynchronous updates.
• Figure 3: FedECADO proposes a multi-rate integration that evaluates the central agent step at intermediate time points by linearly interpolating and extrapolating client states to the synchronized time point.
• Figure 4: The training loss and classification accuracy for a VGG-11 model trained on a CIFAR-10 dataset across 100 clients with non-IID Dirichlet distribution.
• Figure 5: The training loss and classification accuracy of a VGG-11 model trained on a CIFAR-10 dataset across 100 clients where each client's learning rate and number of epochs is randomly determined by \ref{['eq:random_lr']},\ref{['eq:random_epochs']}.

Theorems & Definitions (4)

• Theorem 1
• proof
• Lemma 1
• Lemma 2