On the global complexity of a derivative-free Levenberg-Marquardt algorithm via orthogonal spherical smoothing

TL;DR

The paper addresses derivative-free optimization for nonlinear least squares where the Jacobian is expensive or unavailable. It introduces a derivative-free Levenberg-Marquardt algorithm that uses orthogonal spherical smoothing to construct approximate Jacobians, and defines probabilistic first-order accurate gradient models. Theoretical contributions include explicit bounds showing the gradient estimates are accurate with high probability and a global, high-probability iteration complexity of order $O(\epsilon^{-2})$ to reach $\|\nabla f(x)\|\le\epsilon$. Numerical experiments demonstrate that OSS-based Jacobian estimates can outperform finite-difference implementations on benchmark problems, supporting the practical viability of the proposed approach for large-scale, expensive-Jacobian nonlinear least squares.

Abstract

In this paper, we propose a derivative-free Levenberg-Marquardt algorithm for nonlinear least squares problems, where the Jacobian matrices are approximated via orthogonal spherical smoothing. It is shown that the gradient models which use the approximate Jacobian matrices are probabilistically first-order accurate, and the high probability complexity bound of the algorithm is also given.

Figures (1)

• Figure 1: Performance profile

Theorems & Definitions (28)

• Lemma 2.2
• proof
• Lemma 2.3
• proof
• Lemma 2.4
• proof
• Definition 3.1
• Theorem 3.2
• proof
• Lemma 3.3