Adaptive Federated Learning via Dynamical System Model

TL;DR

This work addresses the sensitivity to hyperparameters in heterogeneous federated learning by reframing FL as a continuous-time dynamical system with a central agent and client states connected by coupling flows. It introduces Adaptive FedECADO, which jointly adapts client learning dynamics and server aggregation through a circuit-inspired design controlled by a single tolerance $\gamma$, leveraging a Thevenin-impedance-based critical damping to set momentum $L_i$ and Backward-Euler-based adaptive time stepping for stability. The method ensures stable convergence despite non-IID data and variable client compute, with LTE-driven time-step adaptation guiding both client and server updates. Empirical results show robust performance across datasets and models, outperforming state-of-the-art adaptive methods that require careful hyperparameter tuning, while incurring modest overhead due to the dynamical-system formulation.

Abstract

Hyperparameter selection is critical for stable and efficient convergence of heterogeneous federated learning, where clients differ in computational capabilities, and data distributions are non-IID. Tuning hyperparameters is a manual and computationally expensive process as the hyperparameter space grows combinatorially with the number of clients. To address this, we introduce an end-to-end adaptive federated learning method in which both clients and central agents adaptively select their local learning rates and momentum parameters. Our approach models federated learning as a dynamical system, allowing us to draw on principles from numerical simulation and physical design. Through this perspective, selecting momentum parameters equates to critically damping the system for fast, stable convergence, while learning rates for clients and central servers are adaptively selected to satisfy accuracy properties from numerical simulation. The result is an adaptive, momentum-based federated learning algorithm in which the learning rates for clients and servers are dynamically adjusted and controlled by a single, global hyperparameter. By designing a fully integrated solution for both adaptive client updates and central agent aggregation, our method is capable of handling key challenges of heterogeneous federated learning, including objective inconsistency and client drift. Importantly, our approach achieves fast convergence while being insensitive to the choice of the global hyperparameter, making it well-suited for rapid prototyping and scalable deployment. Compared to state-of-the-art adaptive methods, our framework is shown to deliver superior convergence for heterogeneous federated learning while eliminating the need for hyperparameter tuning both client and server updates.

Figures (9)

• Figure 1: Simulating a single-variable second-order ODE, $\ddot{x}(t)+L\dot{x}+10x(t)=0$, we vary the parameter $L$ to achieve overdamped, critically damped, and underdamped systems.
• Figure 2: Classification accuracy of Adaptive FedECADO compared to baseline methods across a hyperparameter sweep (via random search for each method) under heterogeneous settings. Results are shown for (a) ResNet-18 on CIFAR-10, (b) ResNet-50 on CIFAR-100.
• Figure 3: The dynamical system model of federated learning can be represented by an electrical circuit. In the circuit representation, each global state, $x_c$, is modeled by a node-voltage connected to multiple nodes whose voltage represents the client states, $x_i$. These nodes are connected by an inductor whose dynamics are modeled by \ref{['eq:inductor_ode']}.
• Figure 4: Equivalent circuit represented of the aggregation step for central agents in \ref{['eq:central_agent_capacitor_gs']}-\ref{['eq:central_agent_inductor_gs']}
• Figure 5: Each client branch in the aggregation circuit (Figure \ref{['fig:aggregation_linear_circuit']}) can be reduced to a series RLC since the Thevenin impedance looking out of each client branch effectively looks like a capacitance for the central agent.