An Adaptive Regularized Proximal Newton-Type Methods for Composite Optimization over the Stiefel Manifold

TL;DR

This work develops two adaptive regularized Newton-type methods for composite optimization on the Stiefel manifold: ARPQN, a proximal quasi-Newton method with adaptive quadratic regularization, and ARPN, a proximal Newton variant using retreated Newton steps. ARPQN achieves global convergence with a guaranteed local linear rate, while ARPN attains global convergence with local superlinear convergence under standard smoothness and Hessian assumptions; both employ line-searched retractions and LBFGS-based Hessian approximations to solve tangent-space subproblems, with ASSN as a practical solver for the subproblems. Empirical results on compressed modes and sparse PCA demonstrate that ARPQN, particularly with SVD retraction, often outperforms existing proximal-Newton-type methods in iteration count and runtime, whereas ARPN shows superior theoretical convergence properties at higher computational cost. The work contributes a rigorous convergence and complexity analysis for manifold-based regularized Newton methods and validates their practical effectiveness in non-Euclidean composite optimization.

Abstract

Recently, the proximal Newton-type method and its variants have been generalized to solve composite optimization problems over the Stiefel manifold whose objective function is the summation of a smooth function and a nonsmooth function. In this paper, we propose an adaptive quadratically regularized proximal quasi-Newton method, named ARPQN, to solve this class of problems. Under some mild assumptions, the global convergence, the local linear convergence rate and the iteration complexity of ARPQN are established. Numerical experiments and comparisons with other state-of-the-art methods indicate that ARPQN is very promising. We also propose an adaptive quadratically regularized proximal Newton method, named ARPN. It is shown the ARPN method has a local superlinear convergence rate under certain reasonable assumptions, which demonstrates attractive convergence properties of regularized proximal Newton methods.

Figures (6)

• Figure 1: Comparison on CM problem, different $n=\{128,256,512,1024,2048\}$ with $r= 10$ and $\mu=0.1$.
• Figure 2: Comparison on CM problem, different $r=\{1,4,8,12,16\}$ with $n=1024$ and $\mu=0.1$.
• Figure 3: Comparison on CM problem, different $\mu=\{0.05,0.10,0.15,0.20,0.25\}$ with $n= 1024$ and $r=10$.
• Figure 4: Comparison on Sparse PCA problem, different $n=\{500, 1000, 1500, 2000, 2500, 3000\}$ with $r= 20$ and $\mu=1.0$.
• Figure 5: Comparison on Sparse PCA problem, different $r=\{5,10,15,20,25\}$ with $n= 2000$ and $\mu=1.0$.

Theorems & Definitions (38)

• Definition 2.1
• Remark 1
• Proposition 2.1
• Definition 2.2
• Lemma 3.1
• Remark 2
• Lemma 3.2
• proof
• Lemma 3.3
• proof