Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Complexity of trust-region methods with unbounded Hessian approximations for smooth and nonsmooth optimization

Geoffroy Leconte, Dominique Orban

TL;DR

This work establishes a sharp worst-case evaluation complexity for trust-region methods solving nonsmooth regularized optimization when Hessian approximations $B_k$ may grow unbounded. By constraining the growth of $\|B_k\|$ in terms of the number of successful iterations via a parameter $p\in[0,1)$, the authors show the bound degrades from the classical $O(\epsilon^{-2})$ to $O(\epsilon^{-2/(1-p)})$, and they prove sharpness through a Hermite-interpolated construction in 1D. The analysis also provides convergence guarantees for limit points and verifies the bound numerically, extending prior results to potentially unbounded Hessians and offering a first step toward Powell's conjecture about exponential iteration complexity in such settings. The results have implications for the design and analysis of trust-region and proximal-trust-region methods in both smooth and nonsmooth, constrained contexts, including quasi-Newton updates that may produce unbounded Hessian growth.

Abstract

We develop a worst-case evaluation complexity bound for trust-region methods in the presence of unbounded Hessian approximations. We use the algorithm of arXiv:2103.15993v3 as a model, which is designed for nonsmooth regularized problems, but applies to unconstrained smooth problems as a special case. Our analysis assumes that the growth of the Hessian approximation is controlled by the number of successful iterations. We show that the best known complexity bound of $ε^{-2}$ deteriorates to $ε^{-2/(1-p)}$, where $0 \le p < 1$ is a parameter that controls the growth of the Hessian approximation. The faster the Hessian approximation grows, the more the bound deteriorates. We construct an objective that satisfies all of our assumptions and for which our complexity bound is attained, which establishes that our bound is sharp. To the best of our knowledge, our complexity result is the first to consider potentially unbounded Hessians and is a first step towards addressing a conjecture of Powell [38] that trust-region methods may require an exponential number of iterations in such a case. Numerical experiments conducted in double precision arithmetic are consistent with the analysis.

Complexity of trust-region methods with unbounded Hessian approximations for smooth and nonsmooth optimization

TL;DR

This work establishes a sharp worst-case evaluation complexity for trust-region methods solving nonsmooth regularized optimization when Hessian approximations may grow unbounded. By constraining the growth of in terms of the number of successful iterations via a parameter , the authors show the bound degrades from the classical to , and they prove sharpness through a Hermite-interpolated construction in 1D. The analysis also provides convergence guarantees for limit points and verifies the bound numerically, extending prior results to potentially unbounded Hessians and offering a first step toward Powell's conjecture about exponential iteration complexity in such settings. The results have implications for the design and analysis of trust-region and proximal-trust-region methods in both smooth and nonsmooth, constrained contexts, including quasi-Newton updates that may produce unbounded Hessian growth.

Abstract

We develop a worst-case evaluation complexity bound for trust-region methods in the presence of unbounded Hessian approximations. We use the algorithm of arXiv:2103.15993v3 as a model, which is designed for nonsmooth regularized problems, but applies to unconstrained smooth problems as a special case. Our analysis assumes that the growth of the Hessian approximation is controlled by the number of successful iterations. We show that the best known complexity bound of deteriorates to , where is a parameter that controls the growth of the Hessian approximation. The faster the Hessian approximation grows, the more the bound deteriorates. We construct an objective that satisfies all of our assumptions and for which our complexity bound is attained, which establishes that our bound is sharp. To the best of our knowledge, our complexity result is the first to consider potentially unbounded Hessians and is a first step towards addressing a conjecture of Powell [38] that trust-region methods may require an exponential number of iterations in such a case. Numerical experiments conducted in double precision arithmetic are consistent with the analysis.
Paper Structure (8 sections, 23 theorems, 105 equations, 2 figures, 1 table, 1 algorithm)

This paper contains 8 sections, 23 theorems, 105 equations, 2 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Context
  3. Background
  4. Models and trust-region algorithm
  5. Convergence and complexity with potentially unbounded Hessians
  6. Sharpness of the complexity bound
  7. Numerical verification of the bound
  8. Discussion

Key Result

Proposition 1

Let $C \subset \mathds{R}^n$ be nonempty and compact, and let asm:models be satisfied. Let eq:nlp satisfy the constraint qualification eq:cq at $x \in C$. Assume $0 \in \mathop{\textup{argmin}}_{s} m(s; x) + \chi(x + s \mid C)$, and let the latter subproblem satisfy the constraint qualification eq:c

Figures (2)

  • Figure 1: TR logs with $\epsilon = 1/3$. outer denotes the iteration number, inner is the number of iterations performed by the subsolver to solve \ref{['eq:model-k']} with the model in \ref{['eq:def-model-quad']}, $\sqrt{}\xi cp / \sqrt{} \nu$ is $\nu_k^{-1/2}\xi_{\mathrm{cp}}(\Delta_k; x_k, \nu_k)^{1/2}$, $\sqrt{}\xi$ is the numerator of \ref{['eq:rhok']}, $\|s\|$ is $\|s_k\|$, and the remaining columns refer unambiguously to data used in \ref{['alg:tr-nonsmooth']}.
  • Figure 2: Illustration of example \ref{['eq:def-f']} with $\epsilon = 1/3$. Top row: values of $f$ (left) and of $f'$ (right) for $x \in [0, x_{k_{\epsilon}+1}]$. Bottom row: iterates $x_k$ (left) and steps $s_k$ (right) for $k \in [0, k_{\epsilon}+1]$.

Theorems & Definitions (44)

  • Proposition 1: leconte-orban-2024
  • Proposition 2: aravkin-baraldi-orban-2022 and aravkin-baraldi-leconte-orban-2023
  • Lemma 1
  • proof
  • Proposition 3
  • proof
  • Proposition 4
  • proof
  • Theorem 1
  • proof
  • ...and 34 more