A Smooth Locally Exact Penalty Method for Optimization Problems over Generalized Stiefel Manifolds
Linshuo Jiang, Nachuan Xiao, Xin Liu
TL;DR
This work addresses optimization with generalized orthogonality constraints $X^\top M X = I_p$ where $M$ may be rank-deficient, by introducing a Smooth Locally Exact Penalty (SLEP) $h(X)$ that couples a smooth surrogate of the Lagrangian with a quadratic penalty on $X^\top M X - I_p$. The authors derive a closed-form gradient and Hessian for $h$ and prove that, for sufficiently large finite $\beta$, FOSP and SOSP of the penalized problem coincide with those of the original constrained problem in a neighborhood of the feasible set; they also establish a Łojasiewicz gradient inequality transfer. Building on this, they propose an unconstrained gradient-based solver (SLBB) for the penalized problem and demonstrate, through extensive numerical experiments on generalized Stiefel-related tasks, that SLBB outperforms state-of-the-art Riemannian BB methods, particularly when $M$ is rank-deficient. The results show substantial reductions in per-iteration cost and robust performance across problem scales, highlighting the practical potential of SLEP for large-scale constrained optimization on generalized Stiefel manifolds.
Abstract
In this paper, we consider a class of optimization problems constrained to the generalized Stiefel manifold. Such problems are fundamental to a wide range of real-world applications, including generalized canonical correlation analysis, linear discriminant analysis, and electronic structure calculations. Existing works mainly focuses on cases where the generalized orthogonality constraint is induced by a symmetric positive definite matrix M, a setting where the geometry essentially reduces to that of the standard Stiefel manifold. However, many practical scenarios involve a singular M, which introduces significant analytical and computational challenges. Therefore, we propose a Smooth Locally Exact Penalty model (SLEP) and establish its equivalence to the original problem in the aspect of stationary points under a finitly large penalty parameter. This penalty model admits the direct application of various unconstrained optimization techniques, with convergence guarantees inherited from established results. Compared to Riemannian optimization approaches, our proposed penalty mode eliminates the need for retractions and vector transports, hence significantly reducing per-iteration computational costs. Extensive numerical experiments validate our theoretical results and demonstrate the effectiveness and practical potential of the proposed penalty model SLEP.