Topology-aware Generalization of Decentralized SGD

TL;DR

The paper develops topology-aware stability and generalization bounds for vanilla D-SGD, showing that the consensus model is $O(N^{-1} + m^{-1} + \\lambda^2)$-stable in expectation under non-convex non-smooth objectives. It derives a corresponding generalization bound in expectation, $O(N^{-(1+\\alpha)/2} + m^{-(1+\\alpha)/2} + \\lambda^{1+\\alpha} + \\phi_\mathcal{S})$, highlighting the critical role of the spectral gap $1-\\lambda$ in generalization performance. The theory explains why more connected topologies and early consensus-distance control improve generalization, and is supported by empirical results on VGG-11 and ResNet-18 across CIFAR-10/100 and Tiny ImageNet. The work fills a gap by providing topology-aware generalization analysis for vanilla D-SGD, contrasting with prior topology-insensitive results for projected variants.

Abstract

This paper studies the algorithmic stability and generalizability of decentralized stochastic gradient descent (D-SGD). We prove that the consensus model learned by D-SGD is $\mathcal{O}{(N^{-1}+m^{-1} +λ^2)}$-stable in expectation in the non-convex non-smooth setting, where $N$ is the total sample size, $m$ is the worker number, and $1+λ$ is the spectral gap that measures the connectivity of the communication topology. These results then deliver an $\mathcal{O}{(N^{-(1+α)/2}+ m^{-(1+α)/2}+λ^{1+α} + φ_{\mathcal{S}})}$ in-average generalization bound, which is non-vacuous even when $λ$ is closed to $1$, in contrast to vacuous as suggested by existing literature on the projected version of D-SGD. Our theory indicates that the generalizability of D-SGD is positively correlated with the spectral gap, and can explain why consensus control in initial training phase can ensure better generalization. Experiments of VGG-11 and ResNet-18 on CIFAR-10, CIFAR-100 and Tiny-ImageNet justify our theory. To our best knowledge, this is the first work on the topology-aware generalization of vanilla D-SGD. Code is available at https://github.com/Raiden-Zhu/Generalization-of-DSGD.

Figures (4)

• Figure 1: Illustration of some commonly-used communication topologies.
• Figure 2: Histograms of the weight differences of the last layers of the ResNet-18 models (1024 dimensions $\times$ 10 classes=10240 parameters) trained by AWC D-SGD on $\mathcal{S}$ and $\mathcal{S}^{(i)}$ that differ by only one data point. Thirty ResNet-18 models are trained on data sampled from the MNIST dataset lecun1998gradient, each model is trained with 16 workers for 3000 iterations.
• Figure 3: Loss differences of training VGG-11 with D-SGD on different topologies.
• Figure C.1: Loss differences of training ResNet-18 with D-SGD on different topologies.

Theorems & Definitions (22)

• Definition 1: Distributed On-average Stability
• Theorem 1
• Corollary 2: Stability in Expectation with $\eta_t\equiv\eta$
• Lemma 3: Generalization via Distributed On-average Stability
• Theorem 4: Generalization Bound in Expectation with $\eta_t\equiv\eta$
• Remark 1
• Corollary 5
• Remark A.1
• Remark A.2
• Definition A.1: Excess Error Decomposition