Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Fast Newton Method Under Local Lipschitz Smoothness

Serge Gratton, Sadok Jerad, Philippe L. Toint

TL;DR

The paper tackles unconstrained nonconvex optimization under a local Lipschitz smoothness condition instead of the global Hessian Lipschitz assumption. It introduces AN2CLS, a fast second-order method that alternates Newton-like steps with negative-curvature steps and uses a Stepcomp procedure to adaptively regularize and choose steps, achieving the optimal first-order rate $O(|\log(\epsilon)|\epsilon^{-3/2})$ under local smoothness. It extends this framework to second-order critical points via SOAN2CLS, providing a corresponding complexity bound and preserving the same adaptive, geometry-aware philosophy. Two practical implementations are explored—AN2CLSE (exact solves) and AN2CLSK (Krylov-based)—with preconditioning discussed—demonstrating competitiveness against standard trust-region and cubic-regularization baselines on CUTEst problems and illustrating the method’s practical potential for large-scale nonconvex optimization.

Abstract

A new, fast second-order method is proposed that achieves the optimal $\mathcal{O}\left(|\log(ε)|ε^{-3/2}\right)$ complexity to obtain first-order $ε$-stationary points. Crucially, this is deduced without assuming the standard global Lipschitz Hessian continuity condition, but only using an appropriate local smoothness requirement. The algorithm exploits Hessian information to compute a Newton step and a negative curvature step when needed, in an approach similar to that of the AN2C method. Inexact versions of the Newton step and negative curvature are proposed in order to reduce the cost of evaluating second-order information. Details are given of such an iterative implementation using Krylov subspaces. An extended algorithm for finding second-order critical points is also developed and its complexity is again shown to be within a log factor of the optimal one. Initial numerical experiments are discussed for both factorised and Krylov variants, which demonstrate the competitiveness of the proposed algorithm.

A Fast Newton Method Under Local Lipschitz Smoothness

TL;DR

The paper tackles unconstrained nonconvex optimization under a local Lipschitz smoothness condition instead of the global Hessian Lipschitz assumption. It introduces AN2CLS, a fast second-order method that alternates Newton-like steps with negative-curvature steps and uses a Stepcomp procedure to adaptively regularize and choose steps, achieving the optimal first-order rate under local smoothness. It extends this framework to second-order critical points via SOAN2CLS, providing a corresponding complexity bound and preserving the same adaptive, geometry-aware philosophy. Two practical implementations are explored—AN2CLSE (exact solves) and AN2CLSK (Krylov-based)—with preconditioning discussed—demonstrating competitiveness against standard trust-region and cubic-regularization baselines on CUTEst problems and illustrating the method’s practical potential for large-scale nonconvex optimization.

Abstract

A new, fast second-order method is proposed that achieves the optimal complexity to obtain first-order -stationary points. Crucially, this is deduced without assuming the standard global Lipschitz Hessian continuity condition, but only using an appropriate local smoothness requirement. The algorithm exploits Hessian information to compute a Newton step and a negative curvature step when needed, in an approach similar to that of the AN2C method. Inexact versions of the Newton step and negative curvature are proposed in order to reduce the cost of evaluating second-order information. Details are given of such an iterative implementation using Krylov subspaces. An extended algorithm for finding second-order critical points is also developed and its complexity is again shown to be within a log factor of the optimal one. Initial numerical experiments are discussed for both factorised and Krylov variants, which demonstrate the competitiveness of the proposed algorithm.
Paper Structure (14 sections, 16 theorems, 62 equations, 2 figures, 2 tables)

This paper contains 14 sections, 16 theorems, 62 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Adaptive Newton with Negative Curvature using Local Smoothness
  3. Complexity Analysis of AN2CLS
  4. Finding Second-Order Critical Points
  5. Step Computation and Preconditioning
  6. Exact Step Computation
  7. Krylov Variant
  8. Preconditioned Variants
  9. Numerical Illustration
  10. Using Exact Linear Solves and Eigenvalue Decomposition
  11. Krylov Variants
  12. Conclusions and Perspectives
  13. Proof of Lemma \ref{['Lipbounds']}
  14. Proof of Theorem \ref{['ComplexityboundSecond']}

Key Result

Lemma 2.1

Let $f$ be three times differentiable and suppose that there exists $M_0\geq 0$ and $M_1 > 0$ such that and that $f$ verifies L0L1smoothrelax. Then, there exits $(L_0,L_1,\delta)$ such that AS.3 holds.

Figures (2)

  • Figure 1: Performance profile of both AN2CLSE and AN2CE on three different set of problems (small, medium, large)
  • Figure 2: Performance profile of both AN2CLSK and AN2CK on three different sets of problems (small, medium, large)

Theorems & Definitions (16)

  • Lemma 2.1
  • Lemma 2.2
  • Lemma 2.3
  • Lemma 2.4
  • Lemma 2.5
  • Lemma 3.1
  • Lemma 3.2
  • Lemma 3.3
  • Lemma 3.4
  • Theorem 3.5
  • ...and 6 more