Dynamic Topology Optimization for Non-IID Data in Decentralized Learning

TL;DR

Morph introduces a fully decentralized topology optimization for decentralized learning that selects incoming neighbors based on local model dissimilarity, maintaining a fixed in-degree while enabling progressive peer discovery. It uses cosine similarity per layer and quasi-transitive inference to estimate dissimilarity with unknown peers, combined with a two-phase, bias-then-random neighbor sampling to preserve connectivity. Empirical results on CIFAR-10 and FEMNIST show Morph consistently surpasses static and epidemic baselines and closely tracks the fully connected upper bound, achieving significant accuracy gains and lower inter-node variance with fewer communication rounds. The work demonstrates that leveraging model dissimilarity as a topology signal can achieve robust, scalable, and efficient decentralized learning under non-IID data without global coordination.

Abstract

Decentralized learning (DL) enables a set of nodes to train a model collaboratively without central coordination, offering benefits for privacy and scalability. However, DL struggles to train a high accuracy model when the data distribution is non-independent and identically distributed (non-IID) and when the communication topology is static. To address these issues, we propose Morph, a topology optimization algorithm for DL. In Morph, nodes adaptively choose peers for model exchange based on maximum model dissimilarity. Morph maintains a fixed in-degree while dynamically reshaping the communication graph through gossip-based peer discovery and diversity-driven neighbor selection, thereby improving robustness to data heterogeneity. Experiments on CIFAR-10 and FEMNIST with up to 100 nodes show that Morph consistently outperforms static and epidemic baselines, while closely tracking the fully connected upper bound. On CIFAR-10, Morph achieves a relative improvement of 1.12x in test accuracy compared to the state-of-the-art baselines. On FEMNIST, Morph achieves an accuracy that is 1.08x higher than Epidemic Learning. Similar trends hold for 50 node deployments, where Morph narrows the gap to the fully connected upper bound within 0.5 percentage points on CIFAR-10. These results demonstrate that Morph achieves higher final accuracy, faster convergence, and more stable learning as quantified by lower inter-node variance, while requiring fewer communication rounds than baselines and no global knowledge.

Figures (7)

• Figure 1: Node $i$ gets connection requests from three requesting nodes $j$, $h$ and $m$ that share their dissimilarity value with $i$. Node $m$ had approximated its dissimilarity with node $i$ using the cosine inequality. Node $i$ select the top-k connection requests (here $k=2$) and uses its new outgoing connections to share its model updates.
• Figure 2: Probability for the communication graph to be connected depending on the number $d_s$ of connections selected using peer dissimilarity and the number $d_r$ of connections selected randomly with different system sizes ($n=100, 1000, 2000$). In experiments, one has to choose $d_r$ and $d_s$ values that minimize $d_r+d_s$ and such that the communication graph is always connected.
• Figure 3: Performance comparison on CIFAR-10 with 100 nodes in a non-IID setting using degree-3 topologies. The panels show: (a) mean top-1 test accuracy over communication rounds (shaded regions denote standard deviation across five runs), (b) mean test loss, and (c) inter-node variance, i.e., the variance of per-node test accuracies across the entire system. Inter-node variance captures fairness and consistency: lower values indicate that nodes converge to similar performance levels. Epidemic Learning (EL) suffers from high inter-node variance ($\approx 15.5$), reflecting severe inconsistency across nodes, while Morph matches the stability of the fully connected topology (variance $<0.02$) at far lower communication cost.
• Figure 4: Test accuracy on CIFAR-10 with $100$ nodes under different connectivity levels ($k=3,7,14$). Morph consistently approaches the performance of the fully connected topology across all connectivities, while Epidemic Learning lags behind, especially at low connectivity. The Static topology reaches competitive accuracy only at $k=7$, but is less stable across other settings. Higher connectivity reduces the performance gap between methods, with Morph maintaining accuracy close to the upper bound.
• Figure 5: Ablation study on the effect of hyperparameters in Morph using CIFAR-10 with $100$ nodes. Left: Impact of the softmax sharpness parameter $\beta$. Right: Impact of the similarity evaluation interval $\Delta_r$. Lower $\beta$ improves learning performance, while values of $\Delta_r < 1000$ have little influence on convergence speed.