Towards Understanding the Generalizability of Delayed Stochastic Gradient Descent

TL;DR

The paper addresses how asynchronous delays in SGD affect generalization, proposing a sharper average-stability framework via generating functions. It derives bounds showing that, under a fixed learning rate, delayed SGD can reduce generalization error for convex quadratic and strongly convex problems, with analogous results for random delays. The authors validate the theory with experiments on LIBSVM datasets and non-convex models, observing reduced overfitting as delay increases. This work provides a data-dependent perspective on delay-generalization tradeoffs and suggests directions for extending stability-based analyses to broader, possibly non-convex, settings.

Abstract

Stochastic gradient descent (SGD) performed in an asynchronous manner plays a crucial role in training large-scale machine learning models. However, the generalization performance of asynchronous delayed SGD, which is an essential metric for assessing machine learning algorithms, has rarely been explored. Existing generalization error bounds are rather pessimistic and cannot reveal the correlation between asynchronous delays and generalization. In this paper, we investigate sharper generalization error bound for SGD with asynchronous delay $τ$. Leveraging the generating function analysis tool, we first establish the average stability of the delayed gradient algorithm. Based on this algorithmic stability, we provide upper bounds on the generalization error of $\tilde{\mathcal{O}}(\frac{T-τ}{nτ})$ and $\tilde{\mathcal{O}}(\frac{1}{n})$ for quadratic convex and strongly convex problems, respectively, where $T$ refers to the iteration number and $n$ is the amount of training data. Our theoretical results indicate that asynchronous delays reduce the generalization error of the delayed SGD algorithm. Analogous analysis can be generalized to the random delay setting, and the experimental results validate our theoretical findings.

Figures (6)

• Figure 1: Generalization error of the ResNet-18 model trained with delayed SGD for classifying the CIFAR-100 data set. The experiment varied only the asynchronous delays and fixed other parameters, where the learning rate $\eta = 0.1$.
• Figure 2: Schematic of the delayed gradient algorithms. While worker $m$ is computing and uploading the gradient, the server has performed $\tau$ asynchronous model updates.
• Figure 3: Experimental results for solving quadratic convex problems with delayed SGD. We test the generalization error of different fixed delays on four LIBSVM datasets rcv1, gisette, covtype and ijcnn1. The variance in the plots is due to selecting different random seeds in multiple trials.
• Figure 4: Generalization error of solving quadratic convex problems by the delayed SGD algorithm with random delays.
• Figure 5: Training and testing loss of solving quadratic convex problems by the delayed SGD algorithm with fixed and random delays.

Theorems & Definitions (16)

• Definition 1: average stability
• Lemma 1
• Remark 1
• Remark 2
• Remark 3
• Proposition 1
• Lemma 2
• Theorem 1
• Theorem 2
• Remark 4