Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Practical Regularized Quasi-Newton Methods with Inexact Function Values

Hiroki Hamaguchi, Naoki Marumo, Akiko Takeda

TL;DR

This work proposes a noise-tolerant regularized quasi-Newton method equipped with a relaxed Armijo-type line search, designed to remain stable under inaccurate function evaluations, and establishes a global convergence rate of $\mathcal{O}(1/\varepsilon^2)$ for reaching a first-order stationary point under the assumed error model.

Abstract

Many practical optimization problems involve objective function values that are corrupted by unavoidable numerical errors. In smooth nonconvex optimization, quasi-Newton methods combined with line search are widely used due to their efficiency and scalability. These methods implicitly assume accurate function evaluations and thus may fail to converge in noisy settings. Developing fast and robust quasi-Newton methods for such scenarios is therefore crucial. To address this issue, we propose a noise-tolerant regularized quasi-Newton method equipped with a relaxed Armijo-type line search, designed to remain stable under inaccurate function evaluations. By combining a regularization parameter update rule inspired by Objective-Function-Free Optimization and the AdaGrad-Norm method, we establish a global convergence rate of $\mathcal{O}(1/\varepsilon^2)$ for reaching a first-order stationary point under the assumed error model. We performed extensive experiments on the CUTEst benchmark collection with artificially noisy objective function evaluations, as well as with low-precision floating-point arithmetic (64-, 32-, and 16-bit). The results demonstrate that the proposed method is substantially more robust than several existing methods, while maintaining competitive practical convergence speed and computational cost.

Practical Regularized Quasi-Newton Methods with Inexact Function Values

TL;DR

This work proposes a noise-tolerant regularized quasi-Newton method equipped with a relaxed Armijo-type line search, designed to remain stable under inaccurate function evaluations, and establishes a global convergence rate of for reaching a first-order stationary point under the assumed error model.

Abstract

Many practical optimization problems involve objective function values that are corrupted by unavoidable numerical errors. In smooth nonconvex optimization, quasi-Newton methods combined with line search are widely used due to their efficiency and scalability. These methods implicitly assume accurate function evaluations and thus may fail to converge in noisy settings. Developing fast and robust quasi-Newton methods for such scenarios is therefore crucial. To address this issue, we propose a noise-tolerant regularized quasi-Newton method equipped with a relaxed Armijo-type line search, designed to remain stable under inaccurate function evaluations. By combining a regularization parameter update rule inspired by Objective-Function-Free Optimization and the AdaGrad-Norm method, we establish a global convergence rate of for reaching a first-order stationary point under the assumed error model. We performed extensive experiments on the CUTEst benchmark collection with artificially noisy objective function evaluations, as well as with low-precision floating-point arithmetic (64-, 32-, and 16-bit). The results demonstrate that the proposed method is substantially more robust than several existing methods, while maintaining competitive practical convergence speed and computational cost.
Paper Structure (31 sections, 11 theorems, 77 equations, 3 figures, 2 tables, 2 algorithms)

This paper contains 31 sections, 11 theorems, 77 equations, 3 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries and Problem Setting
  3. Regularized Quasi-Newton Method
  4. Objective-Function-Free Optimization
  5. Problem Setting
  6. Proposed Algorithm
  7. Computation of the Regularized Quasi-Newton Direction (line \ref{['line:d_k']}):
  8. Relaxed Armijo Condition with an Error-Absorbing Term (line \ref{['line:armijo']}):
  9. Adaptive Update of the Regularization Parameter (lines \ref{['line:mu_0_condition']}--\ref{['line:update_theta']}):
  10. Theoretical Analysis
  11. Assumptions
  12. Error Bound on Function Evaluations
  13. Backtracking Stage
  14. Bound for the case mu k = 0
  15. Bound for the case mu k > 0
  16. ...and 16 more sections

Key Result

Lemma 1

If $x_k$ and $x_{k+1}$ satisfy $f(x_{k+1}) \le f(x_k)$, it holds that

Figures (3)

  • Figure 1: Performance profile for artificial noise setting (level of $10^{-3}$) and $\epsilon_{\mathrm{gtol}}=10^{-2}$.
  • Figure 2: Performance profiles across different floating-point precisions (64-, 32-, and 16-bit) and tolerance levels. The results demonstrate the robustness of the proposed methods.
  • Figure 3: Mean execution times for the experiment in \ref{['sec:comp_time']}. This plot roughly indicates the extra computational overhead of each method. Error bars represent one standard deviation over $100$ runs.

Theorems & Definitions (20)

  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • Proposition 1
  • proof
  • Lemma 4
  • proof
  • ...and 10 more