A Proximal Modified Quasi-Newton Method for Nonsmooth Regularized Optimization
Youssef Diouane, Mohamed Laghdaf Habiboullah, Dominique Orban
TL;DR
The paper introduces R2N, a proximal modified quasi-Newton method for nonsmooth, potentially nonconvex regularized optimization, and its diagonal variant R2DH. It establishes global convergence without requiring Lipschitz continuity of the gradient and permits unbounded model Hessians under a mild growth condition, delivering tight worst-case complexity bounds that extend to $p$-dependent Hessian growth with $0\le p\le1$. Non-monotone variants and diagonal Hessian specializations enhance practical performance, with detailed complexity analysis and robust subproblem strategies. Comprehensive numerical experiments across basis-pursuit denoise, matrix completion, nonlinear SVM, image denoising, and an inverse nonlinear problem demonstrate competitive performance, especially for diagonal and subproblem-solver configurations, and validate the method’s applicability to a broad class of nonsmooth, nonconvex problems.
Abstract
We develop R2N, a modified quasi-Newton method for minimizing the sum of a $\mathcal{C}^1$ function $f$ and a lower semi-continuous prox-bounded $h$. Both $f$ and $h$ may be nonconvex. At each iteration, our method computes a step by minimizing the sum of a quadratic model of $f$, a model of $h$, and an adaptive quadratic regularization term. A step may be computed by a variant of the proximal-gradient method. An advantage of R2N over trust-region (TR) methods is that proximal operators do not involve an extra TR indicator. We also develop the variant R2DH, in which the model Hessian is diagonal, which allows us to compute a step without relying on a subproblem solver when $h$ is separable. R2DH can be used as standalone solver, but also as subproblem solver inside R2N. We describe non-monotone variants of both R2N and R2DH. Global convergence of a first-order stationarity measure to zero holds without relying on local Lipschitz continuity of $\nabla f$, while allowing model Hessians to grow unbounded, an assumption particularly relevant to quasi-Newton models. Under Lipschitz-continuity of $\nabla f$, we establish a tight worst-case complexity bound of $O(1 / ε^{2/(1 - p)})$ to bring said measure below $ε> 0$, where $0 \leq p < 1$ controls the growth of model Hessians. The latter must not diverge faster than $|\mathcal{S}_k|^p$, where $\mathcal{S}_k$ is the set of successful iterations up to iteration $k$. When $p = 1$, we establish the tight exponential complexity bound $O(\exp(c ε^{-2}))$ where $c > 0$ is a constant. We describe our Julia implementation and report numerical experience on a classic basis-pursuit problem, an image denoising problem, a minimum-rank matrix completion problem, a nonlinear support vector machine and an inverse nonlinear problem.