Decentralized Learning Made Practical with Client Sampling

TL;DR

Plexus tackles the problem of deploying decentralized learning on large-scale, heterogeneous edge networks with churn. It introduces a decentralized peer sampler that selects a small subset of online nodes per round and a single per round aggregator to perform sample wide aggregation, enabling global progress without centralized coordination. The paper provides a full system design, formalized sampling, churn handling, parameter guidelines, and an open source implementation, and demonstrates substantial improvements in time to accuracy, communication, and training resources across four tasks on realistic traces, up to 1000 nodes. This approach offers a practical path to scalable, privacy-preserving cross-device learning with robust performance in real-world edge environments.

Abstract

Decentralized learning (DL) leverages edge devices for collaborative model training while avoiding coordination by a central server. Due to privacy concerns, DL has become an attractive alternative to centralized learning schemes since training data never leaves the device. In a round of DL, all nodes participate in model training and exchange their model with some other nodes. Performing DL in large-scale heterogeneous networks results in high communication costs and prolonged round durations due to slow nodes, effectively inflating the total training time. Furthermore, current DL algorithms also assume all nodes are available for training and aggregation at all times, diminishing the practicality of DL. This paper presents Plexus, an efficient, scalable, and practical DL system. Plexus (1) avoids network-wide participation by introducing a decentralized peer sampler that selects small subsets of available nodes that train the model each round and, (2) aggregates the trained models produced by nodes every round. Plexus is designed to handle joining and leaving nodes (churn). We extensively evaluate Plexus by incorporating realistic traces for compute speed, pairwise latency, network capacity, and availability of edge devices in our experiments. Our experiments on four common learning tasks empirically show that Plexus reduces time-to-accuracy by 1.2-8.3x, communication volume by 2.4-15.3x and training resources needed for convergence by 6.4-370x compared to baseline DL algorithms.

Figures (7)

• Figure 1: The effect of churn (left) and the convergence of local and global aggregation (right) in D-PSGD.
• Figure 2: Overview of round $k$ and $k+1$ in Plexus, including 4 total nodes ($n = 4$) and a sample size of 2 ($s = 2$). Nodes 1 and 3 are in sample $S^k$ and first train the aggregated model they received on their local data. They then send their updated model to the aggregator $a^{k+1}$ in sample $S^{k+1}$. When the aggregator receives all updated models, it aggregates the incoming models and forwards them to the participants in the next sample $S^{k+1}$.
• Figure 3: The model convergence (top row), communication volume (middle row), and training resource usage (bottom row) for Plexus, GL and D-PSGD, for four learning tasks. We evaluate D-PSGD both with a one-peer (OP) exponential graph and a $k$-regular graph with $k = 10$.
• Figure 4: Breakdown of network usage by Plexus in the churn scenario, per message type and for each dataset.
• Figure 5: The performance of Plexus on the FEMNIST learning task, for different sample sizes $s$.