Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Method

Ahmed Khaled, Konstantin Mishchenko, Chi Jin

TL;DR

DoWG introduces a universal, parameter-free gradient-descent method for convex optimization that adapts automatically to both Lipschitz and smooth objectives without hyperparameter tuning. It builds on distance-based weighting of gradients, forming a weighted running sum $v_t$ and a pointwise stepsize $\eta_t=\bar{r}_t^2/\sqrt{v_t}$ to achieve convergence matching the best-tuned GD rates up to polylogarithmic factors in both nonsmooth and smooth settings. Theoretical guarantees are complemented by algorithm-independent lemmas and an unconstrained-domain extension, and empirical results on CIFAR-10 demonstrate competitive performance and edge-of-stability dynamics. Overall, DoWG provides a practical, parameter-free alternative to line-search or restart-based methods while preserving strong convergence guarantees across problem geometries.

Abstract

This paper proposes a new easy-to-implement parameter-free gradient-based optimizer: DoWG (Distance over Weighted Gradients). We prove that DoWG is efficient -- matching the convergence rate of optimally tuned gradient descent in convex optimization up to a logarithmic factor without tuning any parameters, and universal -- automatically adapting to both smooth and nonsmooth problems. While popular algorithms following the AdaGrad framework compute a running average of the squared gradients to use for normalization, DoWG maintains a new distance-based weighted version of the running average, which is crucial to achieve the desired properties. To complement our theory, we also show empirically that DoWG trains at the edge of stability, and validate its effectiveness on practical machine learning tasks.

DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Method

TL;DR

DoWG introduces a universal, parameter-free gradient-descent method for convex optimization that adapts automatically to both Lipschitz and smooth objectives without hyperparameter tuning. It builds on distance-based weighting of gradients, forming a weighted running sum and a pointwise stepsize to achieve convergence matching the best-tuned GD rates up to polylogarithmic factors in both nonsmooth and smooth settings. Theoretical guarantees are complemented by algorithm-independent lemmas and an unconstrained-domain extension, and empirical results on CIFAR-10 demonstrate competitive performance and edge-of-stability dynamics. Overall, DoWG provides a practical, parameter-free alternative to line-search or restart-based methods while preserving strong convergence guarantees across problem geometries.

Abstract

This paper proposes a new easy-to-implement parameter-free gradient-based optimizer: DoWG (Distance over Weighted Gradients). We prove that DoWG is efficient -- matching the convergence rate of optimally tuned gradient descent in convex optimization up to a logarithmic factor without tuning any parameters, and universal -- automatically adapting to both smooth and nonsmooth problems. While popular algorithms following the AdaGrad framework compute a running average of the squared gradients to use for normalization, DoWG maintains a new distance-based weighted version of the running average, which is crucial to achieve the desired properties. To complement our theory, we also show empirically that DoWG trains at the edge of stability, and validate its effectiveness on practical machine learning tasks.
Paper Structure (11 sections, 8 theorems, 53 equations, 4 figures, 1 table, 1 algorithm)

This paper contains 11 sections, 8 theorems, 53 equations, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Algorithms and theory
  4. Baselines: gradient descent and normalized gradient descent
  5. DoWG
  6. Experimental results
  7. Algorithm-independent results
  8. Proofs for DoWG
  9. Smooth case
  10. Nonsmooth case
  11. Unconstrained domain extension

Key Result

Theorem 1

Suppose that $f$ is convex with minimizer $x_{\ast}$. Let $f_{\ast} = f(x_{\ast})$. Let $D_0 \overset{\text{def}}{=} \norm{x_0 - x_{\ast}}$ be the initial distance to the optimum. Denote by $\hat{x}_T = \frac{1}{T} \sum_{t=0}^{T-1} x_t$ the average iterate returned by GD. Then:

Figures (4)

  • Figure 1: Two trajectories of gradient descent on the one-dimensional quadratic $f(x) = \frac{L x^2}{2}$, with $L=100$.
  • Figure 2: NGD iterations on $\ell_2$-regularized linear regression on the mushrooms dataset from LibSVM chang11_libsvm with $\eta = 0.1$. Top (a) shows the function suboptimality over time. Observe that as the number of iterations grow, the method becomes non-monotonic. Bottom (b) shows the effective stepsize $\eta_{\mathrm{eff}, t} = \frac{0.1}{\norm{\nabla f(x_t)}}$ over time.
  • Figure 3: DoWG iterations on $\ell_2$-regularized linear regression on the mushrooms dataset from LibSVM chang11_libsvm with $r_{\epsilon} = 10^{-6}$. Top (a) shows the function suboptimality over time. Observe that as the number of iterations grow, the method becomes non-monotonic. Bottom (b) shows the DoWG stepsize over time.
  • Figure 4: VGG11 (top) and ResNet-50 (bottom) training on CIFAR10. Left: test accuracy, middle: train loss, right: step sizes.

Theorems & Definitions (15)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • proof
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • ...and 5 more