Heavy-Tail Phenomenon in Decentralized SGD

TL;DR

The paper analyzes heavy-tailed phenomena in decentralized SGD (DE-SGD) and shows that DE-SGD yields polynomial tails in stationary distributions under mild smoothness and outside-compact strong-convexity assumptions. For quadratic losses, the tail-index $\alpha$ is the unique root of $h(\alpha)=1$, with $h$ depending on step-size $\eta$, batch-size, and the network topology via the mixing matrix; the index increases with batch-size and decreases with step-size (for $\alpha\ge 1$). The authors compare DE-SGD to centralized and disconnected SGD, revealing a nuanced interplay where network structure can either amplify or mitigate tail heaviness depending on parameter regimes, including the number of nodes and connectivity. Numerical experiments on synthetic data and neural networks corroborate the theory, demonstrating heavier tails for DE-SGD relative to C-SGD and context-dependent comparisons with Dis-SGD across topologies and scales.

Abstract

Recent theoretical studies have shown that heavy-tails can emerge in stochastic optimization due to `multiplicative noise', even under surprisingly simple settings, such as linear regression with Gaussian data. While these studies have uncovered several interesting phenomena, they consider conventional stochastic optimization problems, which exclude decentralized settings that naturally arise in modern machine learning applications. In this paper, we study the emergence of heavy-tails in decentralized stochastic gradient descent (DE-SGD), and investigate the effect of decentralization on the tail behavior. We first show that, when the loss function at each computational node is twice continuously differentiable and strongly convex outside a compact region, the law of the DE-SGD iterates converges to a distribution with polynomially decaying (heavy) tails. To have a more explicit control on the tail exponent, we then consider the case where the loss at each node is a quadratic, and show that the tail-index can be estimated as a function of the step-size, batch-size, and the topological properties of the network of the computational nodes. Then, we provide theoretical and empirical results showing that DE-SGD has heavier tails than centralized SGD. We also compare DE-SGD to disconnected SGD where nodes distribute the data but do not communicate. Our theory uncovers an interesting interplay between the tails and the network structure: we identify two regimes of parameters (stepsize and network size), where DE-SGD can have lighter or heavier tails than disconnected SGD depending on the regime. Finally, to support our theoretical results, we provide numerical experiments conducted on both synthetic data and neural networks.

Figures (8)

• Figure 1: The contour plot of $e({\eta,N})=1-2\mathbb{P}\left(\min_{1\leq i\leq N}a_{i}^{2}+\max_{1\leq i\leq N}a_{i}^{2}<2/\eta\right)$.
• Figure 2: The red curve satisfies $\hat{\alpha}=\hat{\alpha}_{dis}$ (up to a $o(\delta)$ term), $d$=$b$=100, $a_i \sim \mathcal{N}(0, I_d)$ with $\hat{\rho}_{dis}=0$ line in orange.
• Figure 3: Illustration of three main cases on synthetic data
• Figure 4: Tail-index $\alpha$ for different setting on MNIST and CIFAR10.
• Figure 5: Illustration of the network architectures.

Theorems & Definitions (26)

• Proposition 1
• Theorem 2
• Theorem 3
• Proposition 4
• Theorem 5
• Corollary 6
• Corollary 7
• Corollary 8
• Theorem 9
• Corollary 10