Resource Utilization Optimized Federated Learning

TL;DR

The paper addresses the dual idle-time bottlenecks in federated learning: server-device task dependency and device stragglers. It proposes FedOptima, a resource-optimized FL system that combines asynchronous device-side training with centralized server-side training, augmented by an auxiliary network on devices and a memory-efficient activation flow managed by a task scheduler. Key contributions include an idle-time analysis, independent device training with balanced scheduling, and a memory-management mechanism enabling scalability to many devices. Empirical results show FedOptima achieving higher or comparable accuracy, 1.9x–21.8x training acceleration, up to 93.9% server idle reduction and 81.8% device idle reduction, and 1.1x–2.0x throughput gains across CNN and transformer workloads on heterogeneous testbeds, indicating strong practical benefits for resource-constrained edge deployments.

Abstract

Federated learning (FL) systems facilitate distributed machine learning across a server and multiple devices. However, FL systems have low resource utilization limiting their practical use in the real world. This inefficiency primarily arises from two types of idle time: (i) task dependency between the server and devices, and (ii) stragglers among heterogeneous devices. This paper introduces FedOptima, a resource-optimized FL system designed to simultaneously minimize both types of idle time; existing systems do not eliminate or reduce both at the same time. FedOptima offloads the training of certain layers of a neural network from a device to server using three innovations. First, devices operate independently of each other using asynchronous aggregation to eliminate straggler effects, and independently of the server by utilizing auxiliary networks to minimize idle time caused by task dependency. Second, the server performs centralized training using a task scheduler that ensures balanced contributions from all devices, improving model accuracy. Third, an efficient memory management mechanism on the server increases scalability of the number of participating devices. Four state-of-the-art offloading-based and asynchronous FL methods are chosen as baselines. Experimental results show that compared to the best results of the baselines on convolutional neural networks and transformers on multiple lab-based testbeds, FedOptima (i) achieves higher or comparable accuracy, (ii) accelerates training by 1.9x to 21.8x, (iii) reduces server and device idle time by up to 93.9% and 81.8%, respectively, and (iv) increases throughput by 1.1x to 2.0x.

Figures (13)

• Figure 1: Training timeline for various federated learning (FL) methods with one server and two devices (Device 1 is assumed to be faster than Device 2). For simplicity, only one transfer of the model and activations per device is shown for FedOptima.
• Figure 2: Communication volume per round.
• Figure 3: Memory required on server for varying no. of devices.
• Figure 4: Overview of FedOptima.
• Figure 5: The activation flow control between a device and the server. A device only sends activations to the server when the Sender is active.