Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Affine-Invariant Global Non-Asymptotic Convergence Analysis of BFGS under Self-Concordance

Qiujiang Jin, Aryan Mokhtari

TL;DR

This work establishes affine-invariant, non-asymptotic global convergence guarantees for the BFGS quasi-Newton method when optimizing a strictly convex, twice-differentiable function that is $M$-strongly self-concordant. By leveraging Armijo-Wolfe line search and weighted reformulations, the authors derive global linear and global superlinear rates that depend only on the initial suboptimality and the self-concordant constant, not on strong convexity or Lipschitz gradient/Hessian assumptions. A key contribution is proving that, after a finite iteration index, the unit step becomes admissible and yields a $(C/t)^t$-type superlinear rate, while preserving affine invariance across linear changes of variables. The theory is complemented by numerical experiments on hard cubic and logistic regression objectives, confirming the predicted convergence behavior and the affine-invariance property. These results extend global non-asymptotic convergence theory for BFGS to a broad, affine-invariant class of self-concordant functions and offer practical implications for line-search-based quasi-Newton optimization.

Abstract

In this paper, we establish global non-asymptotic convergence guarantees for the BFGS quasi-Newton method without requiring strong convexity or the Lipschitz continuity of the gradient or Hessian. Instead, we consider the setting where the objective function is strictly convex and strongly self-concordant. For an arbitrary initial point and any arbitrary positive-definite initial Hessian approximation, we prove global linear and superlinear convergence guarantees for BFGS when the step size is determined using a line search scheme satisfying the weak Wolfe conditions. Moreover, all our global guarantees are affine-invariant, with the convergence rates depending solely on the initial error and the strongly self-concordant constant. Our results extend the global non-asymptotic convergence theory of BFGS beyond traditional assumptions and, for the first time, establish affine-invariant convergence guarantees aligning with the inherent affine invariance of the BFGS method.

Affine-Invariant Global Non-Asymptotic Convergence Analysis of BFGS under Self-Concordance

TL;DR

This work establishes affine-invariant, non-asymptotic global convergence guarantees for the BFGS quasi-Newton method when optimizing a strictly convex, twice-differentiable function that is -strongly self-concordant. By leveraging Armijo-Wolfe line search and weighted reformulations, the authors derive global linear and global superlinear rates that depend only on the initial suboptimality and the self-concordant constant, not on strong convexity or Lipschitz gradient/Hessian assumptions. A key contribution is proving that, after a finite iteration index, the unit step becomes admissible and yields a -type superlinear rate, while preserving affine invariance across linear changes of variables. The theory is complemented by numerical experiments on hard cubic and logistic regression objectives, confirming the predicted convergence behavior and the affine-invariance property. These results extend global non-asymptotic convergence theory for BFGS to a broad, affine-invariant class of self-concordant functions and offer practical implications for line-search-based quasi-Newton optimization.

Abstract

In this paper, we establish global non-asymptotic convergence guarantees for the BFGS quasi-Newton method without requiring strong convexity or the Lipschitz continuity of the gradient or Hessian. Instead, we consider the setting where the objective function is strictly convex and strongly self-concordant. For an arbitrary initial point and any arbitrary positive-definite initial Hessian approximation, we prove global linear and superlinear convergence guarantees for BFGS when the step size is determined using a line search scheme satisfying the weak Wolfe conditions. Moreover, all our global guarantees are affine-invariant, with the convergence rates depending solely on the initial error and the strongly self-concordant constant. Our results extend the global non-asymptotic convergence theory of BFGS beyond traditional assumptions and, for the first time, establish affine-invariant convergence guarantees aligning with the inherent affine invariance of the BFGS method.

Paper Structure

This paper contains 22 sections, 22 theorems, 145 equations, 10 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Preliminaries
  3. Assumptions
  4. Definitions
  5. Global Linear Convergence Rates
  6. Global Superlinear Convergence Rates
  7. Complexity Analysis
  8. Numerical Experiments
  9. Conclusions
  10. Notations
  11. Lemmas and Propositions
  12. Proof of Lemmas, Propositions and Theorems
  13. Proof of Proposition \ref{['BFGS_affine']}
  14. Proof of Theorem \ref{['thm:first_linear_rate']}
  15. Proof of Theorem \ref{['thm:second_linear_rate']}
  16. ...and 7 more sections

Key Result

Proposition 2.1

Let the iterations $\{x_t\}_{t=0}^{+\infty}$ be generated by the BFGS algorithm applied to the objective function $f(x)$, as defined in QN_update-eq:BFGS_update. Consider the iterates $\{\dot{x}_t\}_{t=0}^{+\infty}$ produced by applying BFGS to the transformed function $\phi(x) = f(Ax)$, where $A \i

Figures (10)

  • Figure 1: Convergence rates of BFGS with different $B_0$, gradient descent and accelerated gradient descent for solving the hard cubic function with different dimensions.
  • Figure 2: Convergence rates of BFGS with different $B_0$, gradient descent and accelerated gradient descent for solving the logistic regression function with different dimensions.
  • Figure 3: Step size of BFGS with different $B_0$ using inexact line search for solving the hard cubic function with different dimensions.
  • Figure 4: Convergence rates of BFGS with different $B_0$, gradient descent and accelerated gradient descent for solving the hard cubic function with transformation matrix $A$.
  • Figure 5: Convergence rates of BFGS with different $B_0$, gradient descent and accelerated gradient descent for solving the hard cubic function with different dimensions.
  • ...and 5 more figures

Theorems & Definitions (35)

  • Proposition 2.1
  • Remark 2.1
  • Theorem 3.1
  • Corollary 3.2
  • Theorem 3.3
  • Corollary 3.4
  • Lemma 4.1
  • Lemma 4.2
  • Theorem 4.3
  • Corollary 4.4
  • ...and 25 more