An inexact $q$-order regularized proximal Newton method for nonconvex composite optimization
Ruyu Liu, Shaohua Pan, Yitian Qian
TL;DR
This work tackles nonconvex composite optimization of the form $F(x)=f(x)+g(x)$ with a smooth part $f$ and a nonsmooth convex part $g$. It introduces an inexact $q$-order regularized proximal Newton method for $q\in[2,3]$, unifying with cubic regularization when $q=3$, and proves full convergence to (second-order) stationary points under KL assumptions, achieving local $Q$-superlinear rates that depend on the KL exponent and a local Hölderian error bound. The method constructs a $q$-order majorization and solves subproblems inexactly, with a novel iterate-selection rule that guarantees descent and convergence; global complexity is shown to be $O(\epsilon^{-q/(q-1)})$ under mild regularity. Numerical experiments using ZeroFPR as the inner solver demonstrate practical efficiency on problems with highly nonlinear $f$, including ℓ1-regularized logistic regression, nonconvex Student's $t$ regression, and portfolio optimization with higher moments. The results indicate that the proposed inexact CR framework can achieve fast convergence without relying on strong convexity, offering a robust tool for large-scale nonconvex, nonsmooth optimization in imaging, statistics, and finance.
Abstract
This paper concerns the composite problem of minimizing the sum of a twice continuously differentiable function $f$ and a nonsmooth convex function. For this class of nonconvex and nonsmooth problems, by leveraging a practical inexactness criterion and a novel selection strategy for iterates, we propose an inexact $q$-order regularized proximal Newton method for $q\in[2,3]$, which becomes an inexact cubic regularization (CR) method for $q=3$. We prove that the whole iterate sequence converges to a stationary point for the KL objective function; and when the objective function has the KL property of exponent $θ\in(0,\frac{q-1}{q})$, the convergence has a local $Q$-superlinear rate of order $\frac{q-1}{θq}$. In particular, under a local Hölderian error bound of order $γ\in(\frac{1}{q-1},1]$ on a second-order stationary point set, we show that the iterate and objective value sequences converge to a second-order stationary point and a second-order stationary value, respectively, with a local $Q$-superlinear rate of order $γ(q\!-\!1)$, specified as the $Q$-quadratic rate for $q=3$ and $γ=1$. This is the first practical inexact CR method with $Q$-quadratic convergence rate for nonconvex composite optimization. We validate the efficiency of the CR method with ZeroFPR as the inner solver by applying it to composite optimization problems with highly nonlinear $f$.