Superlinear Optimization Algorithms

TL;DR

The paper addresses efficient minimization of nonlinear objectives, noting that Hessian singularities complicate Newton-type methods. It introduces optimal-control-inspired optimization algorithms that approximate the optimal state trajectory of a related OCP, producing update rules such as $x_{k+1}=x_k - g_k$ with $g_k$ defined via gradients and Hessians. Four variants (Algorithms I–IV) achieve superlinear convergence, with some avoiding Hessian inverses altogether or substituting diagonal Hessian approximations, under suitable convexity and spectral conditions. Numerical experiments illustrate faster convergence than gradient descent and robustness to Hessian singularities, suggesting practical impact for large-scale nonlinear optimization in engineering and data science.

Abstract

This paper proposes several novel optimization algorithms for minimizing a nonlinear objective function. The algorithms are enlightened by the optimal state trajectory of an optimal control problem closely related to the minimized objective function. They are superlinear convergent when appropriate parameters are selected as required. Unlike Newton's method, all of them can be also applied in the case of a singular Hessian matrix. More importantly, by reduction, some of them avoid calculating the inverse of the Hessian matrix or an identical dimension matrix and some of them need only the diagonal elements of the Hessian matrix. In these cases, these algorithms still outperform the gradient descent method. The merits of the proposed optimization algorithm are illustrated by numerical experiments.

Theorems & Definitions (13)

• Lemma 1
• proof
• Lemma 2
• proof
• Lemma 3
• proof
• Theorem 1
• proof
• Theorem 2
• proof