Towards Dynamic Resource Allocation and Client Scheduling in Hierarchical Federated Learning: A Two-Phase Deep Reinforcement Learning Approach

TL;DR

This work tackles dynamic resource allocation and client scheduling in energy-harvesting hierarchical federated learning (HFL). It proposes a two-phase deep deterministic policy gradient framework (TP-DDPG) that splits decisions into two groups: phase 1 uses a DDPG actor-critic to select participating clients and configure their CPU frequencies and transmit powers, while phase 2 uses the Straggler-Aware Client Association and Bandwidth Allocation (SCABA) to optimize client associations and bandwidth, providing rewards to train the DDPG. The objective is $U=\sum_{t=1}^{RR_1} O_t - RT_g$ with $O_t=\lambda|\Omega^t|-T^t$, balancing learning delay and accuracy; the approach reduces the learnable action space and improves convergence. Experimental results on MNIST and CIFAR-10 show TP-DDPG achieves up to 39.4% reduction in training time to reach 0.9 accuracy on MNIST and attains 0.93 accuracy on CIFAR-10, outperforming baselines. This enables scalable, energy-aware HFL at the network edge with real-time adaptation to wireless channels and energy arrivals.

Abstract

Federated learning (FL) is a viable technique to train a shared machine learning model without sharing data. Hierarchical FL (HFL) system has yet to be studied regrading its multiple levels of energy, computation, communication, and client scheduling, especially when it comes to clients relying on energy harvesting to power their operations. This paper presents a new two-phase deep deterministic policy gradient (DDPG) framework, referred to as ``TP-DDPG'', to balance online the learning delay and model accuracy of an FL process in an energy harvesting-powered HFL system. The key idea is that we divide optimization decisions into two groups, and employ DDPG to learn one group in the first phase, while interpreting the other group as part of the environment to provide rewards for training the DDPG in the second phase. Specifically, the DDPG learns the selection of participating clients, and their CPU configurations and the transmission powers. A new straggler-aware client association and bandwidth allocation (SCABA) algorithm efficiently optimizes the other decisions and evaluates the reward for the DDPG. Experiments demonstrate that with substantially reduced number of learnable parameters, the TP-DDPG can quickly converge to effective polices that can shorten the training time of HFL by 39.4% compared to its benchmarks, when the required test accuracy of HFL is 0.9.

Figures (11)

• Figure 1: Illustration of the HFL system.
• Figure 2: The time interval of the $t$-th edge aggregation.
• Figure 3: The structure of the proposed TP-DDPG algorithm, where a DDPG agent consisting of actor-critic networks and an experience replay memory makes decisions on client selection, power control and CPU configuration, and the SCABA decides on client association and bandwidth allocation. The SCABA can be interpreted as part of the environment for the DDPG, and produces a reward for each decision that the DDPG agent makes on client selection, transmit power, and CPU configuration.
• Figure 4: Learning delay versus the average number of scheduled clients per edge aggregation round.
• Figure 5: System utility $U$ as the number of episode grows when $\lambda =0.35$.