Complexity of trust-region methods in the presence of unbounded Hessian approximations

TL;DR

This work extends trust-region complexity analyses to unconstrained optimization with unbounded Hessian models, addressing practical quasi-Newton scenarios. It introduces a broad family of trust-region methods with radius controlled by gradient and Hessian norms, parameterized by $\alpha$ and $\beta$, and analyzes two Hessian-growth regimes: $\|B_k\| = O(|\mathcal{S}_{k-1}|^p)$ and $\|B_k\| = O(k^p)$ for $0\le p\le1$. The authors derive sharp worst-case bounds: for $0\le p<1$, $O([(1-p)\epsilon^{-2}]^{1/(1-p)})$ iterations to reach an $\epsilon$-stationary point, and for $p=1$, an exponential bound $O(\exp(c\epsilon^{-2}))$, with improved results in convex and strongly convex cases. They also provide new sharpness constructions requiring both successful and unsuccessful iterations and validate the framework with numerical experiments, linking theory to practical quasi-Newton implementations and clarifying Powell's historical intuition about complexity growth.

Abstract

We extend traditional complexity analyses of trust-region methods for unconstrained, possibly nonconvex, optimization. Whereas most complexity analyses assume uniform boundedness of the model Hessians, we work with potentially unbounded model Hessians. Boundedness is not guaranteed in practical implementations, in particular ones based on quasi-Newton updates such as PSB, BFGS and SR1. We examine two regimes of Hessian growth: one bounded by a power of the number of successful iterations, and one bounded by a power of the number of iterations. This allows us to formalize and address the intuition of Powell [IMA J. Numer. Ana. 30(1):289-301,2010], who studied convergence under a special case of our assumptions, but whose proof contained complexity arguments. Specifically, for \(0 \leq p < 1\), we establish sharp \(O([(1-p)ε^{-2}]^{1/(1-p)})\) evaluation complexity to find an \(ε\)-stationary point when model Hessians are \(O(|\mathcal{S}_{k-1}|^p)\), where \(|\mathcal{S}_{k-1}|\) is the number of iterations where the step was accepted, up to iteration \(k-1\). For \(p = 1\), which is the case studied by Powell, we establish a sharp \(O(\exp(c_1ε^{-2}))\) evaluation complexity for a certain constant \(c_1 > 0\). This is far better than the double exponential bound that \citet{powell-2010} suspected, and is far worse than other bounds surmised elsewhere in the literature. We establish similar sharp bounds when model Hessians are \(O(k^p)\), where \(k\) is the iteration counter, for \(0 \leq p < 1\). When \(p = 1\), the complexity bound depends on the parameters of the family, but reduces to \(O((1 - \log(ε))\exp(c_2ε^{-2}))\) for a certain constant \(c_2 > 0\) for the special case of the standard trust-region method. As special cases, we derive novel complexity bounds for (strongly) convex objectives under the same growth assumptions.