When Less is More: Achieving Faster Convergence in Distributed Edge Machine Learning

TL;DR

This work addresses the slow convergence and high communication costs of distributed ML on heterogeneous edge devices. Hermes introduces a probabilistic, communication-efficient framework that transmits only major improvements, using test-loss based gating (HermesGUP) and a dynamic, per-worker data/batch allocation guided by a dual-binary search. It also weights gradient aggregation by loss-aware signals and employs prefetching and model compression to further reduce latency and memory usage. Experimental results on MNIST and CIFAR-10 demonstrate substantial speedups (up to 13.22x) and reduced communication (62.1%) while maintaining competitive accuracy, highlighting Hermes as a practical approach for edge DML with straggler mitigation and heterogeneity handling.

Abstract

Distributed Machine Learning (DML) on resource-constrained edge devices holds immense potential for real-world applications. However, achieving fast convergence in DML in these heterogeneous environments remains a significant challenge. Traditional frameworks like Bulk Synchronous Parallel and Asynchronous Stochastic Parallel rely on frequent, small updates that incur substantial communication overhead and hinder convergence speed. Furthermore, these frameworks often employ static dataset sizes, neglecting the heterogeneity of edge devices and potentially leading to straggler nodes that slow down the entire training process. The straggler nodes, i.e., edge devices that take significantly longer to process their assigned data chunk, hinder the overall training speed. To address these limitations, this paper proposes Hermes, a novel probabilistic framework for efficient DML on edge devices. This framework leverages a dynamic threshold based on recent test loss behavior to identify statistically significant improvements in the model's generalization capability, hence transmitting updates only when major improvements are detected, thereby significantly reducing communication overhead. Additionally, Hermes employs dynamic dataset allocation to optimize resource utilization and prevents performance degradation caused by straggler nodes. Our evaluations on a real-world heterogeneous resource-constrained environment demonstrate that Hermes achieves faster convergence compared to state-of-the-art methods, resulting in a remarkable $13.22$x reduction in training time and a $62.1\%$ decrease in communication overhead.

Figures (12)

• Figure 1: Training and communication time being displayed for BSP, SSP (s = 2, refer Sec. \ref{['sec:ssp']}), ASP and EBSP; each line partition highlights a barrier where gradients of that node is pushed to the PS, with worker$_{2}$ and worker$_{3}$ as the slowest and fastest workers respectively.
• Figure 2: Comprehensive breakup of the time taken by workers during one local training cycle on MNIST dataset using CNN. The x-axis represents various node families to which our workers belong, refer to Table \ref{['table:4']} for node configuration.
• Figure 3: Loss fluctuations and failure to converge due to oscillation of global gradients in a cluster of nodes; cluster is trained on MNIST dataset using a CNN.
• Figure 4: Most nodes complete their training in under 2.5 seconds. The overall iteration time in BSP depends on the straggler nodes.
• Figure 5: Wastage of compute due to waiting for stragglers in BSP.