Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Homogeneous Second-Order Descent Framework: A Fast Alternative to Newton-Type Methods

Chang He, Yuntian Jiang, Chuwen Zhang, Dongdong Ge, Bo Jiang, Yinyu Ye

TL;DR

The paper develops a generalized homogeneous framework for second-order descent (HSODF) that replaces Newton steps with symmetric eigenvalue problems, enabling robust performance on ill-conditioned problems. It introduces GHMs with adaptive parameters, demonstrating how classical second-order methods such as trust-region and gradient-regularized Newton steps can be recovered within HSODF, and analyzes per-iteration costs and convergence properties. Two concrete realizations are developed: an adaptive HSODM with global $O(\epsilon^{-3/2})$ complexity for nonconvex problems with second-order Lipschitz Hessians, and a homotopy HSODM with global linear convergence under concordant Lipschitz assumptions. Numerical experiments on CUTEst and high-dimensional logistic regression illustrate competitive performance and the practical viability of warm-start strategies. The framework offers a unifying, efficient path to a range of second-order methods, with potential extensions to nonconvex conic optimization and beyond.

Abstract

This paper proposes a homogeneous second-order descent framework (HSODF) for nonconvex and convex optimization based on the generalized homogeneous model (GHM). In comparison to the Newton steps, the GHM can be solved by extremal symmetric eigenvalue procedures and thus grant an advantage in ill-conditioned problems. Moreover, GHM extends the ordinary homogeneous model (OHM) (Zhang et al. 2022) to allow adaptiveness in the construction of the aggregated matrix. Consequently, HSODF is able to recover some well-known second-order methods, such as trust-region methods and gradient regularized methods, while maintaining comparable iteration complexity bounds. We also study two specific realizations of HSODF. One is adaptive HSODM, which has a parameter-free $O(ε^{-3/2})$ global complexity bound for nonconvex second-order Lipschitz continuous objective functions. The other one is homotopy HSODM, which is proven to have a global linear rate of convergence without strong convexity. The efficiency of our approach to ill-conditioned and high-dimensional problems is justified by some preliminary numerical results.

Homogeneous Second-Order Descent Framework: A Fast Alternative to Newton-Type Methods

TL;DR

The paper develops a generalized homogeneous framework for second-order descent (HSODF) that replaces Newton steps with symmetric eigenvalue problems, enabling robust performance on ill-conditioned problems. It introduces GHMs with adaptive parameters, demonstrating how classical second-order methods such as trust-region and gradient-regularized Newton steps can be recovered within HSODF, and analyzes per-iteration costs and convergence properties. Two concrete realizations are developed: an adaptive HSODM with global complexity for nonconvex problems with second-order Lipschitz Hessians, and a homotopy HSODM with global linear convergence under concordant Lipschitz assumptions. Numerical experiments on CUTEst and high-dimensional logistic regression illustrate competitive performance and the practical viability of warm-start strategies. The framework offers a unifying, efficient path to a range of second-order methods, with potential extensions to nonconvex conic optimization and beyond.

Abstract

This paper proposes a homogeneous second-order descent framework (HSODF) for nonconvex and convex optimization based on the generalized homogeneous model (GHM). In comparison to the Newton steps, the GHM can be solved by extremal symmetric eigenvalue procedures and thus grant an advantage in ill-conditioned problems. Moreover, GHM extends the ordinary homogeneous model (OHM) (Zhang et al. 2022) to allow adaptiveness in the construction of the aggregated matrix. Consequently, HSODF is able to recover some well-known second-order methods, such as trust-region methods and gradient regularized methods, while maintaining comparable iteration complexity bounds. We also study two specific realizations of HSODF. One is adaptive HSODM, which has a parameter-free global complexity bound for nonconvex second-order Lipschitz continuous objective functions. The other one is homotopy HSODM, which is proven to have a global linear rate of convergence without strong convexity. The efficiency of our approach to ill-conditioned and high-dimensional problems is justified by some preliminary numerical results.
Paper Structure (50 sections, 55 theorems, 209 equations, 14 figures, 3 tables, 5 algorithms)

This paper contains 50 sections, 55 theorems, 209 equations, 14 figures, 3 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Connections to prior works
  3. Contributions
  4. Notations
  5. The Homogeneous Second-Order Descent Framework
  6. Motivation for the HSODF
  7. Comparison of the per-iteration cost between HSODF and Newton-type methods
  8. Further numerics of the per-iteration cost
  9. Hilbert matrix and a random right-hand-side
  10. LIBSVM instances
  11. Characterization of primal-dual solutions
  12. Basic results
  13. Continuity of auxiliary functions
  14. Using GHMs in a SOM
  15. Recovering trust-region methods
  16. ...and 35 more sections

Key Result

Lemma 2.1

$[v_k; t_k]$ is the optimal solution of the subproblem eq.ghqm if and only if there exists a dual variable $\theta_k \ge 0$ such that

Figures (14)

  • Figure 1: $g_k \not\perp \mathcal{S}_1$
  • Figure 2: $g_k \perp \mathcal{S}_1$
  • Figure 4: $\|g_k\| = 5.17\times 10^{-1}$
  • Figure 5: $\|g_k\| = 5.17\times 10^{-7}$
  • Figure 7: Convex case
  • ...and 9 more figures

Theorems & Definitions (111)

  • Lemma 2.1: Optimality condition
  • Theorem 2.1
  • Lemma 2.2: Existence of negative curvature
  • Lemma 2.3: Sufficiency in convex case
  • Lemma 2.4: Upper bound of $\theta_k$
  • Lemma 2.5: Lemma 3.1, 3.2, rojas_new_2001
  • Corollary 2.1
  • Lemma 2.6: Ordering of $\tilde{\alpha}_1$
  • Definition 2.1: Auxiliary functions of $\delta_k$
  • Lemma 2.7: Continuity of $\theta_k, \omega_k$
  • ...and 101 more