Energy and Memory-Efficient Federated Learning With Ordered Layer Freezing

TL;DR

This work addresses the challenge of training deep models via Federated Learning on resource-constrained IoT devices while preserving privacy. It introduces FedOLF, which employs Ordered Layer Freezing to keep low-level layers frozen across training rounds, thereby shortening the backpropagation path and reducing memory usage, and combines this with TOA (Tensor Operation Approximation) to reduce communication by sparsifying frozen layers with a norm-based sampling scheme. Theoretical convergence is established under standard non-convex assumptions with a bound on gradient divergence from freezing, and TOA is shown to maintain accuracy while lowering energy and memory costs. Empirical results across EMNIST, CIFAR-10/100, and CINIC-10 demonstrate FedOLF achieving higher accuracy than existing resource-constrained FL methods (with notable improvements on non-iid data) and improved energy efficiency and memory footprint, making FedOLF a practical choice for edge deployments with limited resources.

Abstract

Federated Learning (FL) has emerged as a privacy-preserving paradigm for training machine learning models across distributed edge devices in the Internet of Things (IoT). By keeping data local and coordinating model training through a central server, FL effectively addresses privacy concerns and reduces communication overhead. However, the limited computational power, memory, and bandwidth of IoT edge devices pose significant challenges to the efficiency and scalability of FL, especially when training deep neural networks. Various FL frameworks have been proposed to reduce computation and communication overheads through dropout or layer freezing. However, these approaches often sacrifice accuracy or neglect memory constraints. To this end, in this work, we introduce Federated Learning with Ordered Layer Freezing (FedOLF). FedOLF consistently freezes layers in a predefined order before training, significantly mitigating computation and memory requirements. To further reduce communication and energy costs, we incorporate Tensor Operation Approximation (TOA), a lightweight alternative to conventional quantization that better preserves model accuracy. Experimental results demonstrate that over non-iid data, FedOLF achieves at least 0.3%, 6.4%, 5.81%, 4.4%, 6.27% and 1.29% higher accuracy than existing works respectively on EMNIST (with CNN), CIFAR-10 (with AlexNet), CIFAR-100 (with ResNet20 and ResNet44), and CINIC-10 (with ResNet20 and ResNet44), along with higher energy efficiency and lower memory footprint.

Figures (16)

• Figure 1: A comparison between (a) Random Layer Freezing and (b) Ordered Layer Freezing. The former requires more memory space to pass the gradient information back towards low-level active layers.
• Figure 2: The required memory space of random and ordered layer freezing (Model: ResNet20, Dataset: CIFAR-100).
• Figure 3: During training, the $l$ frozen layers will generate a feature representation $\boldsymbol{x'}_{l}$ that diverges from the true $\boldsymbol{x}_{l}$. Affected by $\boldsymbol{x'}_{l}$, the following active layers also generate inaccurate representations.
• Figure 4: A high-level flow chart demonstrating how FedOLF works. For simplicity, we use an exemplary network consisting of a global model with five layers, and three clients $k_{1}, k_{2}$ and $k_{3}$ with three degrees of system capacity.
• Figure 5: An illustration of the layer-wise aggregation scheme in FedOLF (same as in cocoflfedsl). $W_{l}$ and $W_{k,l}$ respectively stand for the $l$-th layer in the global model and the local model of client $k$. Here we have $1\leq l \leq 5$ and $k \in \{k_{1}, k_{2}, k_{3}\}$.