Krylov Cubic Regularized Newton: A Subspace Second-Order Method with Dimension-Free Convergence Rate

TL;DR

The paper addresses the high computational burden of second-order methods by introducing Krylov CRN, a subspace cubic regularized Newton method that updates within the Krylov subspace generated by the Hessian and gradient. By using Lanczos to obtain an m-dimensional basis, the method preserves the fast CRN convergence while achieving a dimension-free rate of \\mathcal{O}(\\frac{1}{mk} + \\frac{1}{k^2})\\ in the convex setting and a linear rate under strong convexity, with per-iteration cost \\(\\mathcal{O}(md)\\). The convergence analysis hinges on the spectral properties of the Hessian via the quantity \\rho^{(m)}(\\mathbf{H},\\mathbf{g})\\, which can be significantly smaller than general Lipschitz constants in structured problems. The approach also recovers full CRN rates under favorable Hessian spectra and demonstrates superior empirical performance over random-subspace methods on high-dimensional logistic regression tasks, highlighting practical impact for scalable second-order optimization.

Abstract

Second-order optimization methods, such as cubic regularized Newton methods, are known for their rapid convergence rates; nevertheless, they become impractical in high-dimensional problems due to their substantial memory requirements and computational costs. One promising approach is to execute second-order updates within a lower-dimensional subspace, giving rise to subspace second-order methods. However, the majority of existing subspace second-order methods randomly select subspaces, consequently resulting in slower convergence rates depending on the problem's dimension $d$. In this paper, we introduce a novel subspace cubic regularized Newton method that achieves a dimension-independent global convergence rate of ${O}\left(\frac{1}{mk}+\frac{1}{k^2}\right)$ for solving convex optimization problems. Here, $m$ represents the subspace dimension, which can be significantly smaller than $d$. Instead of adopting a random subspace, our primary innovation involves performing the cubic regularized Newton update within the Krylov subspace associated with the Hessian and the gradient of the objective function. This result marks the first instance of a dimension-independent convergence rate for a subspace second-order method. Furthermore, when specific spectral conditions of the Hessian are met, our method recovers the convergence rate of a full-dimensional cubic regularized Newton method. Numerical experiments show our method converges faster than existing random subspace methods, especially for high-dimensional problems.

Figures (2)

• Figure 1: Numerical comparisons of CRN, SSCN, and our proposed Krylov CRN method on a logistic regression problem. Here, $m$ denotes the dimension of the subspace, $n$ is the number of samples, and $d$ is the number of parameters. See Section \ref{['sec:numerical']} for full details.
• Figure 2: Numerical comparisons of CRN, SSCN, and our proposed Krylov CRN method for logistic regression on LIBSVM datasets. Here, $m$ is the subspace dimension, $n$ is the number of samples, and $d$ is the number of parameters. The first row depicts the loss curves in terms of the number of iterations, while the second row depicts these curves in terms of the overall computation time. We observe that Krylov CRN converges much faster than the others when the dimension $d$ is large.

Theorems & Definitions (25)

• Proposition 2.1
• Proposition 3.1
• Proposition 3.2
• Remark 3.1: Computational cost of Algorithm \ref{['alg:subspace_cubic']}
• Lemma 4.1
• Lemma 4.2
• Lemma 4.3
• Theorem 4.4
• Theorem 4.5
• Remark 4.1