Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AdaBB: Adaptive Barzilai-Borwein Method for Convex Optimization

Danqing Zhou, Shiqian Ma, Junfeng Yang

TL;DR

AdaBB, an adaptive gradient method based on the Barzilai-Borwein stepsize is proposed, and it essentially provides a convergent variant of the Barzilai-Borwein method for general convex optimization problems.

Abstract

In this paper, we propose AdaBB, an adaptive gradient method based on the Barzilai-Borwein stepsize. The algorithm is line-search-free and parameter-free, and essentially provides a convergent variant of the Barzilai-Borwein method for general unconstrained convex optimization. We analyze the ergodic convergence of the objective function value and the convergence of the iterates for solving general unconstrained convex optimization. Compared with existing works along this line of research, our algorithm gives the best lower bounds on the stepsize and the average of the stepsizes. Moreover, we present an extension of the proposed algorithm for solving composite optimization where the objective function is the summation of a smooth function and a nonsmooth function. Our numerical results also demonstrate very promising potential of the proposed algorithms on some representative examples.

AdaBB: Adaptive Barzilai-Borwein Method for Convex Optimization

TL;DR

AdaBB, an adaptive gradient method based on the Barzilai-Borwein stepsize is proposed, and it essentially provides a convergent variant of the Barzilai-Borwein method for general convex optimization problems.

Abstract

In this paper, we propose AdaBB, an adaptive gradient method based on the Barzilai-Borwein stepsize. The algorithm is line-search-free and parameter-free, and essentially provides a convergent variant of the Barzilai-Borwein method for general unconstrained convex optimization. We analyze the ergodic convergence of the objective function value and the convergence of the iterates for solving general unconstrained convex optimization. Compared with existing works along this line of research, our algorithm gives the best lower bounds on the stepsize and the average of the stepsizes. Moreover, we present an extension of the proposed algorithm for solving composite optimization where the objective function is the summation of a smooth function and a nonsmooth function. Our numerical results also demonstrate very promising potential of the proposed algorithms on some representative examples.
Paper Structure (11 sections, 28 theorems, 93 equations, 7 figures, 3 tables, 3 algorithms)

This paper contains 11 sections, 28 theorems, 93 equations, 7 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Our AdaBB Algorithm
  3. The General Version of AdaBB and Convergence Analysis
  4. Improved Lower Bound for
  5. Extensions
  6. When is Locally Strongly Convex
  7. Composite Convex Optimization Problems
  8. Numerical Experiments
  9. Logistic Regression
  10. Subproblem of cubic regularized Newton method
  11. Concluding Remarks

Key Result

Lemma 3.1

In both gen-AdaBB-alpha-1 and gen-AdaBB-alpha-2, the stepsize $\alpha_k$ provided by (Option II) is less than or equal to that provided by (Option I).

Figures (7)

  • Figure 1: Results for the logistic regression problem via AdaBB, AdaBB1, AdaBB2, AdaBB3 concerning the function value residual.
  • Figure 2: Results for the logistic regression problem via GD, AdGD, AdaPGM, and AdaBB concerning the function value residual and gradient norm.
  • Figure 3: Results for the logistic regression problem via line search methods, and AdaBB concerning the function value residual.
  • Figure 4: Stepsizes generated by AdGD and AdaBB.
  • Figure 5: Results for the cubic regulation problem for AdaBB, AdaBB1, AdaBB2, AdaBB3 concerning the function value residual.
  • ...and 2 more figures

Theorems & Definitions (60)

  • Lemma 3.1
  • proof
  • Lemma 3.2
  • proof
  • Remark 3.1
  • Lemma 3.3
  • proof
  • Lemma 3.4
  • proof
  • Corollary 3.1
  • ...and 50 more