Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Statistical and Computational Complexities of BFGS Quasi-Newton Method for Generalized Linear Models

Qiujiang Jin, Tongzheng Ren, Nhat Ho, Aryan Mokhtari

TL;DR

It is demonstrated that the iterates of BFGS converge linearly to the optimal solution of the population least-square loss function, and the contraction coefficient of the BFGS algorithm is comparable to that of Newton's method.

Abstract

The gradient descent (GD) method has been used widely to solve parameter estimation in generalized linear models (GLMs), a generalization of linear models when the link function can be non-linear. In GLMs with a polynomial link function, it has been shown that in the high signal-to-noise ratio (SNR) regime, due to the problem's strong convexity and smoothness, GD converges linearly and reaches the final desired accuracy in a logarithmic number of iterations. In contrast, in the low SNR setting, where the problem becomes locally convex, GD converges at a slower rate and requires a polynomial number of iterations to reach the desired accuracy. Even though Newton's method can be used to resolve the flat curvature of the loss functions in the low SNR case, its computational cost is prohibitive in high-dimensional settings as it is $\mathcal{O}(d^3)$, where $d$ the is the problem dimension. To address the shortcomings of GD and Newton's method, we propose the use of the BFGS quasi-Newton method to solve parameter estimation of the GLMs, which has a per iteration cost of $\mathcal{O}(d^2)$. When the SNR is low, for GLMs with a polynomial link function of degree $p$, we demonstrate that the iterates of BFGS converge linearly to the optimal solution of the population least-square loss function, and the contraction coefficient of the BFGS algorithm is comparable to that of Newton's method. Moreover, the contraction factor of the linear rate is independent of problem parameters and only depends on the degree of the link function $p$. Also, for the empirical loss with $n$ samples, we prove that in the low SNR setting of GLMs with a polynomial link function of degree $p$, the iterates of BFGS reach a final statistical radius of $\mathcal{O}((d/n)^{\frac{1}{2p+2}})$ after at most $\log(n/d)$ iterations.

Statistical and Computational Complexities of BFGS Quasi-Newton Method for Generalized Linear Models

TL;DR

It is demonstrated that the iterates of BFGS converge linearly to the optimal solution of the population least-square loss function, and the contraction coefficient of the BFGS algorithm is comparable to that of Newton's method.

Abstract

The gradient descent (GD) method has been used widely to solve parameter estimation in generalized linear models (GLMs), a generalization of linear models when the link function can be non-linear. In GLMs with a polynomial link function, it has been shown that in the high signal-to-noise ratio (SNR) regime, due to the problem's strong convexity and smoothness, GD converges linearly and reaches the final desired accuracy in a logarithmic number of iterations. In contrast, in the low SNR setting, where the problem becomes locally convex, GD converges at a slower rate and requires a polynomial number of iterations to reach the desired accuracy. Even though Newton's method can be used to resolve the flat curvature of the loss functions in the low SNR case, its computational cost is prohibitive in high-dimensional settings as it is , where the is the problem dimension. To address the shortcomings of GD and Newton's method, we propose the use of the BFGS quasi-Newton method to solve parameter estimation of the GLMs, which has a per iteration cost of . When the SNR is low, for GLMs with a polynomial link function of degree , we demonstrate that the iterates of BFGS converge linearly to the optimal solution of the population least-square loss function, and the contraction coefficient of the BFGS algorithm is comparable to that of Newton's method. Moreover, the contraction factor of the linear rate is independent of problem parameters and only depends on the degree of the link function . Also, for the empirical loss with samples, we prove that in the low SNR setting of GLMs with a polynomial link function of degree , the iterates of BFGS reach a final statistical radius of after at most iterations.
Paper Structure (18 sections, 14 theorems, 160 equations, 9 figures)

This paper contains 18 sections, 14 theorems, 160 equations, 9 figures.

Table of Contents

  1. Introduction
  2. BFGS algorithm
  3. Generalized linear model with polynomial link function
  4. Convergence analysis in the low SNR regime: Population loss
  5. Comparison with Newton's method
  6. Convergence analysis in the low SNR regime: Finite sample setting
  7. Numerical experiments
  8. Conclusions
  9. Proofs
  10. Proof of Theorem \ref{['theorem_1']}
  11. Proof of Theorem \ref{['theorem_2']}
  12. Proof of Theorem \ref{['theorem_3']}
  13. Elaboration of Remark \ref{['remark_1']}
  14. Proof of Theorem \ref{['theorem:statistical_rate_BFGS']}
  15. Induction base
  16. ...and 3 more sections

Key Result

Theorem 1

Consider the BFGS method in BFGS_update-difference. Suppose Assumptions ass_1 and ass_2 hold, and the initial Hessian inverse approximation matrix is selected as $H_0 = \nabla^{2}{f(\theta_0)}^{-1}$, where $\theta_0 \in \mathbb{R}^d$ is an arbitrary initial point. If the step size is $\eta_k = 1$ fo where the contraction factors $r_k\in[0,1)$ satisfy

Figures (9)

  • Figure 1: Convergence of factors $\{r_k\}_{k = 0}^{\infty}$ to $r_*$.
  • Figure 2: Convergence of Newton's method, BFGS, GD with constant step size, GD with Polyak step size and mirror descent with constant step size for different values of $d$ and $q$. In plot (a), $m = 100$ and $\eta = 10^{-4}$. In plot (b), $m = 100$ and $\eta = 10^{-8}$. In plot (c), $m = 2000$ and $\eta = 10^{-12}$. In plot (d), $m = 2000$ and $\eta = 10^{-15}$.
  • Figure 3: Convergence of different methods when $d=4$ in high SNR (a) and low SNR (b) regimes. Illustration of the statistical radius of BFGS in high SNR (c) and low SNR (d) regimes.
  • Figure 4: Convergence of different methods $d = 50$ in high SNR (a) and low SNR (b) regimes. Statistical radius of BFGS in high SNR (c) and low SNR (d) settings.
  • Figure 5: Convergence of different methods with $d = 100$ for high SNR regime are shown in (a) and low SNR regime in (b). Convergence of different methods with $d = 500$ for high SNR regime are shown in (c) and low SNR regime in (d).
  • ...and 4 more figures

Theorems & Definitions (27)

  • Remark 1
  • Theorem 1
  • Theorem 2
  • Remark 2
  • Remark 3
  • Theorem 3
  • Theorem 4
  • Lemma 1
  • proof
  • Lemma 2
  • ...and 17 more