Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Interval Hessian-based line-search method for unconstrained nonconvex optimization

Ashutosh Sharma, Gauransh Dingwani, Nikhil Gupta, Vaishnavi Gupta, Ishan Bajaj

TL;DR

This work tackles unconstrained nonconvex optimization by introducing INTHOP, an interval Hessian-based line-search method that guarantees descent. It replaces the exact Hessian with a positive-definite surrogate built from an $\alpha$-shifted interval Hessian of $\mathcal{L}$, and uses interval arithmetic to bound the discrepancy with the true Hessian. The approach combines three eigenvalue-bounding methods (GGN, EM, MK) with fixed or adaptive interval sizes (INTHOP:F, INTHOP:A; A1,A2 variants) to compute descent directions efficiently and selectively recompute Hessians. Empirical results on diverse problems show that INTHOP often requires fewer function/gradient evaluations and substantially fewer $\mathcal{O}(n^3)$ operations than Newton-based methods, while outperforming steepest descent and BFGS in convergence and robustness for nonconvex problems.

Abstract

Second-order Newton-type algorithms that leverage the exact Hessian or its approximation are central to solving nonlinear optimization problems. These algorithms have been proven to achieve a faster convergence rate than the first-order methods and can find second-order stationary points. However, their applications in solving large-scale nonconvex problems are hindered by three primary challenges: (1) the high computational cost associated with Hessian evaluations, (2) its inversion, and (3) ensuring descent direction at points where the Hessian becomes indefinite. We propose INTHOP, an interval Hessian-based optimization algorithm for nonconvex problems. Specifically, we propose a new search direction guaranteed to be descent and requiring Hessian evaluations and inversion only at specific iterations. The proposed search direction is based on approximating the original Hessian matrix by a positive-definite matrix. We prove that the difference between the approximate and exact Hessian is bounded within an interval. Accordingly, the approximate Hessian matrix is reused if the iterates are in the interval while computing the gradients at each iteration. We develop various algorithm variants based on the interval size updating methods and minimum eigenvalue computation methods. We apply the algorithm to an extensive set of test problems and compare its performance with steepest descent, quasi-Newton, and the Newton methods. We show empirically that our method solves more problems in fewer function and gradient evaluations than steepest descent and the quasi-Newton method. Compared to the Newton method, we illustrate that for nonconvex problems, we require substantially less O(n3) operations.

An Interval Hessian-based line-search method for unconstrained nonconvex optimization

TL;DR

This work tackles unconstrained nonconvex optimization by introducing INTHOP, an interval Hessian-based line-search method that guarantees descent. It replaces the exact Hessian with a positive-definite surrogate built from an -shifted interval Hessian of , and uses interval arithmetic to bound the discrepancy with the true Hessian. The approach combines three eigenvalue-bounding methods (GGN, EM, MK) with fixed or adaptive interval sizes (INTHOP:F, INTHOP:A; A1,A2 variants) to compute descent directions efficiently and selectively recompute Hessians. Empirical results on diverse problems show that INTHOP often requires fewer function/gradient evaluations and substantially fewer operations than Newton-based methods, while outperforming steepest descent and BFGS in convergence and robustness for nonconvex problems.

Abstract

Second-order Newton-type algorithms that leverage the exact Hessian or its approximation are central to solving nonlinear optimization problems. These algorithms have been proven to achieve a faster convergence rate than the first-order methods and can find second-order stationary points. However, their applications in solving large-scale nonconvex problems are hindered by three primary challenges: (1) the high computational cost associated with Hessian evaluations, (2) its inversion, and (3) ensuring descent direction at points where the Hessian becomes indefinite. We propose INTHOP, an interval Hessian-based optimization algorithm for nonconvex problems. Specifically, we propose a new search direction guaranteed to be descent and requiring Hessian evaluations and inversion only at specific iterations. The proposed search direction is based on approximating the original Hessian matrix by a positive-definite matrix. We prove that the difference between the approximate and exact Hessian is bounded within an interval. Accordingly, the approximate Hessian matrix is reused if the iterates are in the interval while computing the gradients at each iteration. We develop various algorithm variants based on the interval size updating methods and minimum eigenvalue computation methods. We apply the algorithm to an extensive set of test problems and compare its performance with steepest descent, quasi-Newton, and the Newton methods. We show empirically that our method solves more problems in fewer function and gradient evaluations than steepest descent and the quasi-Newton method. Compared to the Newton method, we illustrate that for nonconvex problems, we require substantially less O(n3) operations.
Paper Structure (28 sections, 2 theorems, 46 equations, 11 figures, 7 algorithms)

This paper contains 28 sections, 2 theorems, 46 equations, 11 figures, 7 algorithms.

Table of Contents

  1. Introduction
  2. Notation
  3. Methodology
  4. Search Direction Construction
  5. Interval Hessian
  6. Interval Hessian Illustrative example
  7. alpha Calculation methods
  8. Approximate Gerschgorin theorem (GGN) for interval matrices
  9. E-Matrix (EM) Method
  10. Mori-Kokame (MK) Method
  11. Algorithm Structure
  12. INTHOP:F
  13. INTHOP:A
  14. INTHOP:A1
  15. INTHOP:A2
  16. ...and 13 more sections

Key Result

Theorem 1

Let $f: \mathbb{R}^n \rightarrow \mathbb{R}$ be a twice-differentiable continuous function and $\mathcal{L}(\mathbf{x})$ be $\alpha$BB convex underestimator constructed on an interval of width $\delta$ such that $\mathbf{x}_t, \mathbf{x}_k\in [\mathbf{x}^L, \mathbf{x}^U]$. Suppose $f$, $\nabla f$, a Suppose the minimum eigenvalue of $\nabla^2 f(\mathbf{x})$ for $x\in[\mathbf{x}^L, \mathbf{x}^U]$ i

Figures (11)

  • Figure 1: Graphical illustration of $f(\mathbf{x}),\ m_k^{N}(\mathbf{x})\ \text{and} \ m_k^{IH}(\mathbf{x})$
  • Figure 2: Graphical illustration of INTHOP with fixed interval size (INTHOP:F) and Algorithm \ref{['algo:SymInt']} used to generate intervals.
  • Figure 3: Paths taken by steepest descent, quasi-Newton, and INTHOP:F on illustrative example 1.
  • Figure 4: (A) Comparison of paths take by INTHOP:F and NEWTON:CAMI on illustrative example 2. (B) Function value as a function of time for INTHOP:F and NEWTON:CAMI.
  • Figure 5: Fraction of problems solved by INTHOP:F variants with (A) gradient evaluations, (B) Hessian evaluations, (C) $\mathcal{O}(n^3)$ operations
  • ...and 6 more figures

Theorems & Definitions (5)

  • Theorem 1
  • proof
  • Lemma 1
  • proof
  • proof