Scalable Decentralized Learning with Teleportation

TL;DR

This work addresses the scaling limitations of decentralized SGD when the number of nodes is large and the network topology induces slow consensus. It introduces Teleportation, a method that activates a small subset of nodes, transfers parameters from the previous active set, updates via SGD, and performs gossip averaging on a compact active topology, thereby decoupling convergence from the total node count. Theoretical results show that Teleportation can eliminate the degradation in convergence rate caused by increasing $n$ and can achieve rates comparable to the best fixed-topology designs, with an efficient parameter-free hyperparameter search that requires only $2T$ iterations. Empirical results on synthetic data and neural networks demonstrate faster convergence and greater stability under heterogeneity, highlighting Teleportation’s practical impact for scalable decentralized learning in data centers and over Internet-connected networks.

Abstract

Decentralized SGD can run with low communication costs, but its sparse communication characteristics deteriorate the convergence rate, especially when the number of nodes is large. In decentralized learning settings, communication is assumed to occur on only a given topology, while in many practical cases, the topology merely represents a preferred communication pattern, and connecting to arbitrary nodes is still possible. Previous studies have tried to alleviate the convergence rate degradation in these cases by designing topologies with large spectral gaps. However, the degradation is still significant when the number of nodes is substantial. In this work, we propose TELEPORTATION. TELEPORTATION activates only a subset of nodes, and the active nodes fetch the parameters from previous active nodes. Then, the active nodes update their parameters by SGD and perform gossip averaging on a relatively small topology comprising only the active nodes. We show that by activating only a proper number of nodes, TELEPORTATION can completely alleviate the convergence rate degradation. Furthermore, we propose an efficient hyperparameter-tuning method to search for the appropriate number of nodes to be activated. Experimentally, we showed that TELEPORTATION can train neural networks more stably and achieve higher accuracy than Decentralized SGD.

Figures (7)

• Figure 1: Illustration of Alg. \ref{['algorithm:simple_version_proposed_method']} with $n=8$ and $k=3$. We use a line as the topology consisting of active nodes $\mathcal{G}_k = (\{1,2,3\}, \{(1,1), (1,2), (2,2), (2,3), (3,3)\})$. The black nodes represent active nodes, and the number written on the node is $\texttt{token\_id}^{(t)}_i$. The blue nodes represent the next active nodes, and the number on the node is $\texttt{token\_id}^{(t+1)}_i$. The first and third graphs from the left represent the communication in line 12, and the other graphs represent the communication in line 8.
• Figure 2: Convergence of the error to the target accuracy $0.001$ for different stochastic noise $\sigma^2$ and heterogeneity $\zeta^2$. We plotted $\frac{1}{k} \sum_{v_i \in V^{(t)}_\text{active}} \| {\bm{x}}_i^{(t)} - {\bm{x}}^\star \|^2$ and $\frac{1}{n} \sum_{i=1}^n \| {\bm{x}}_i^{(t)} - {\bm{x}}^\star \|^2$ as the error for Teleportation and Decentralized SGD, respectively. Teleportation consistently reached the target accuracy faster than Decentralized SGD.
• Figure 3: Test accuracy of Teleportation and Decentralized SGD for different heterogeneity. All methods achieved competitive accuracy in the homogeneous case, while Teleportation outperformed Decentralized SGD in the heterogeneous case.
• Figure 4: Test accuracy of Teleportation and Decentralized SGD under the heterogeneous networks with $\tau=5$. Decentralized SGD with the ring reached a high accuracy faster than the other methods in the homogeneous case, while Teleportation reached a high accuracy faster in the heterogeneous case. Note that the number of epochs was set the same for all methods.
• Figure 5: Illustration of Alg. \ref{['algorithm:proposed_method']} with $n=8$ and $k=3$. We use a line as the topology consisting of active nodes $\mathcal{G}_k = (\{1,2,3\}, \{(1,1), (1,2), (2,2), (2,3), (3,3)\})$. The black nodes represent active nodes, and the number written on the node is $\texttt{token\_id}^{(t)}_i$. The blue nodes represent the next active nodes, and the number on the node is $\texttt{token\_id}^{(t+1)}_i$.

Theorems & Definitions (27)

• Proposition 1: koloskova2020unified
• Theorem 1
• Theorem 2
• Lemma 1
• Theorem 3
• Lemma 2
• Lemma 3
• Lemma 4
• proof
• Lemma 5