Cutting Some Slack for SGD with Adaptive Polyak Stepsizes

TL;DR

This paper considers the family of SPS (Stochastic gradient with a Polyak Stepsize) adaptive methods, and shows that SPS and its recent variants can all be seen as extensions of the Passive-Aggressive methods applied to nonlinear problems.

Abstract

Tuning the step size of stochastic gradient descent is tedious and error prone. This has motivated the development of methods that automatically adapt the step size using readily available information. In this paper, we consider the family of SPS (Stochastic gradient with a Polyak Stepsize) adaptive methods. These are methods that make use of gradient and loss value at the sampled points to adaptively adjust the step size. We first show that SPS and its recent variants can all be seen as extensions of the Passive-Aggressive methods applied to nonlinear problems. We use this insight to develop new variants of the SPS method that are better suited to nonlinear models. Our new variants are based on introducing a slack variable into the interpolation equations. This single slack variable tracks the loss function across iterations and is used in setting a stable step size. We provide extensive numerical results supporting our new methods and a convergence theory.

Figures (7)

• Figure 1: Comparison of the proposed and related methods on different computer vision problems in terms of both error in a validation set and loss in the train set. With the same set of parameters, the new slack variants are competitive throughout different datasets and models. In terms of validation error, SPSL1 achieves the best result in the MNIST and arrives at a close second on the CIFAR-10.
• Figure 2: The real line of possible values for $s_0$ divided into three segments.
• Figure 3: Relative suboptimality ($\ell(w^T)/\ell(w^0)$ y-axis) for each method when using a fixed regularization $reg$ (x-axis) (Figures (a) and (b)) or a fixed $\lambda$ (Figures (c) and (d)). Each method was given a budget of 100 epochs on logistic regression.
• Figure 4: Relative suboptimality ($\ell(w^T)/\ell(w^0)$) for each method with slack parameter $\lambda$ (x-axis) after given a budget of 100 epochs on logistic regression. For the top row of figures, we used a regularization of $reg = 0.1$. For the bottom row, we used a regularization of $reg = 10^{-5}$. The data sets used from left to right are colon-cancer, mushrooms, phishing, and cod-rna. We find consistently that for problems far from interpolation, $\lambda$ small works best, while for problems close to interpolatiom $\lambda$ large works best.
• Figure 5: Relative suboptimality ($\ell(w^T)/\ell(w^0)$) for each method when using regularization $reg$ (x-axis) after given a budget of 100 epochs on logistic regression. On the top row, all methods used $\lambda =1.$ On the bottom row, all methods used $\lambda = 0.01.$ The data sets used from left to right are colon-cancer, mushrooms, phishing, and cod-rna.

Theorems & Definitions (65)

• lemma 1
• proof
• lemma 2
• proof
• lemma 3
• lemma 4
• theorem 5
• theorem 6
• theorem 7
• lemma 8