Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Provably Faster Gradient Descent via Long Steps

Benjamin Grimmer

TL;DR

This work establishes new convergence guarantees for gradient descent in smooth convex optimization via a computer-assisted analysis technique and shows that long steps, which may increase the objective value in the short term, lead to provably faster convergence in the long term.

Abstract

This work establishes new convergence guarantees for gradient descent in smooth convex optimization via a computer-assisted analysis technique. Our theory allows nonconstant stepsize policies with frequent long steps potentially violating descent by analyzing the overall effect of many iterations at once rather than the typical one-iteration inductions used in most first-order method analyses. We show that long steps, which may increase the objective value in the short term, lead to provably faster convergence in the long term. A conjecture towards proving a faster $O(1/T\log T)$ rate for gradient descent is also motivated along with simple numerical validation.

Provably Faster Gradient Descent via Long Steps

TL;DR

This work establishes new convergence guarantees for gradient descent in smooth convex optimization via a computer-assisted analysis technique and shows that long steps, which may increase the objective value in the short term, lead to provably faster convergence in the long term.

Abstract

This work establishes new convergence guarantees for gradient descent in smooth convex optimization via a computer-assisted analysis technique. Our theory allows nonconstant stepsize policies with frequent long steps potentially violating descent by analyzing the overall effect of many iterations at once rather than the typical one-iteration inductions used in most first-order method analyses. We show that long steps, which may increase the objective value in the short term, lead to provably faster convergence in the long term. A conjecture towards proving a faster rate for gradient descent is also motivated along with simple numerical validation.
Paper Structure (12 sections, 8 theorems, 52 equations, 2 figures, 1 table)

This paper contains 12 sections, 8 theorems, 52 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Sketch of Proof Technique - Reducing proving eventual descent to an SDP
  3. Straightforward Stepsize Patterns
  4. Convergence Guarantees for Straightforward Stepsize Patterns
  5. Faster Convergence for Straightforward Patterns given Growth Bounds
  6. Certificates of Straightforwardness
  7. The Worst-Case Value Function is Upper Bounded by an SDP
  8. An SDP Feasibility Certificate that implies Straightforwardness
  9. Certificates of Straightforwardness Proving Guarantees in Table \ref{['tab:patterns-and-rates']}
  10. Future Directions
  11. Acknowledgements.
  12. A Computer-Generated Straightforwardness Certificate with $\Delta= 10^{-5}$ and $\epsilon=10^{-9}$ for $h=(1.5, 2.2, 1.5, 12.0, 1.5, 2.2, 1.5)$

Key Result

Lemma 2.1

For any $L,D>0$, $p_{L,D}$ is concave.

Figures (2)

  • Figure 1: Least squares problems minimizing $\|Ax-b\|^2_2$ (left) and $\|Ax-b\|^2_2 + \|x\|^2_2$ (right) with i.i.d. normal entries in $A\in\mathbb{R}^{n\times n}$ and $b\in\mathbb{R}^n$ for $n=4000$. Objective gaps are plotted over $T=2000$ iterations with $h=(1)$ and with each pattern from Table \ref{['tab:patterns-and-rates']}. Note this second objective is substantially more strongly convex, so its faster linear convergence is expected. For comparison, Nesterov's accelerated method (without modification to utilize strong convexity) is shown.
  • Figure 2: For unconstrained, constrained, and composite minimization, the stepsize patterns satisfying a natural generalization of straightforwardness are shown. Patterns $(h_1,h_2)$ of length two and symmetric patterns $(h_1,h_2,h_1)$ of length three were sampled at every $0.1$ increment. Straightforwardness was approximated by solving a performance estimation problem determining if $f(x_t) - p_\star$ (or $f(x_t) + r(x_t)- p_\star$ for composite problems) is always at most $f(x_0) - p_\star - \frac{\sum h_i}{LD^2} (f(x_0) - p_\star)^2$ (or $f(x_0) +r(x_0)- p_\star - \frac{\sum h_i}{LD^2} (f(x_0) +r(x_0)- p_\star)^2$) given the initial gap was at most $\Delta = 10^{-4}$.

Theorems & Definitions (16)

  • Conjecture 1.1
  • Lemma 2.1
  • proof
  • Theorem 2.1
  • proof
  • Theorem 2.2
  • proof
  • Lemma 3.1
  • Theorem 3.1
  • proof
  • ...and 6 more