Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A line search framework with restarting for noisy optimization problems

Albert S. Berahas, Michael J. O'Neill, Clément W. Royer

TL;DR

The paper tackles noisy unconstrained optimization by introducing a restart-enabled line-search framework that maintains worst-case iteration and evaluation guarantees even when function and gradient evaluations are noisy. By adaptively selecting step sizes through a noise-aware line search and applying restart conditions to switch between descent directions and gradient steps, the method accommodates both nonlinear conjugate gradient and quasi-Newton updates. The main theoretical contribution is a complexity bound on the number of iterations needed to achieve an $\varepsilon$-stationary point, highlighting how restart frequency and noise levels influence convergence. Empirically, restarted variants (NLCGr, LBFGSr) exhibit robustness and competitive performance on CUTEst problems across noise regimes, with LBFGS-based restart often delivering the strongest results while NLCG variants show lower sensitivity to restarts.

Abstract

Nonlinear optimization methods are typically iterative and make use of gradient information to determine a direction of improvement and function information to effectively check for progress. When this information is corrupted by noise, designing a convergent and practical algorithmic process becomes challenging, as care must be taken to avoid taking bad steps due to erroneous information. For this reason, simple gradient-based schemes have been quite popular, despite being outperformed by more advanced techniques in the noiseless setting. In this paper, we propose a general algorithmic framework based on line search that is endowed with iteration and evaluation complexity guarantees even in a noisy setting. These guarantees are obtained as a result of a restarting condition, that monitors desirable properties for the steps taken at each iteration and can be checked even in the presence of noise. Experiments using a nonlinear conjugate gradient variant and a quasi-Newton variant illustrate that restarting can be performed without compromising practical efficiency and robustness.

A line search framework with restarting for noisy optimization problems

TL;DR

The paper tackles noisy unconstrained optimization by introducing a restart-enabled line-search framework that maintains worst-case iteration and evaluation guarantees even when function and gradient evaluations are noisy. By adaptively selecting step sizes through a noise-aware line search and applying restart conditions to switch between descent directions and gradient steps, the method accommodates both nonlinear conjugate gradient and quasi-Newton updates. The main theoretical contribution is a complexity bound on the number of iterations needed to achieve an -stationary point, highlighting how restart frequency and noise levels influence convergence. Empirically, restarted variants (NLCGr, LBFGSr) exhibit robustness and competitive performance on CUTEst problems across noise regimes, with LBFGS-based restart often delivering the strongest results while NLCG variants show lower sensitivity to restarts.

Abstract

Nonlinear optimization methods are typically iterative and make use of gradient information to determine a direction of improvement and function information to effectively check for progress. When this information is corrupted by noise, designing a convergent and practical algorithmic process becomes challenging, as care must be taken to avoid taking bad steps due to erroneous information. For this reason, simple gradient-based schemes have been quite popular, despite being outperformed by more advanced techniques in the noiseless setting. In this paper, we propose a general algorithmic framework based on line search that is endowed with iteration and evaluation complexity guarantees even in a noisy setting. These guarantees are obtained as a result of a restarting condition, that monitors desirable properties for the steps taken at each iteration and can be checked even in the presence of noise. Experiments using a nonlinear conjugate gradient variant and a quasi-Newton variant illustrate that restarting can be performed without compromising practical efficiency and robustness.

Paper Structure

This paper contains 20 sections, 5 theorems, 52 equations, 11 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem and algorithm
  3. Algorithmic framework
  4. Algorithmic instances
  5. Nonlinear conjugate gradient
  6. Quasi-Newton methods
  7. Complexity analysis
  8. Assumptions and decrease guarantees
  9. Main results
  10. Numerical experiments
  11. Setup
  12. Test problems
  13. Algorithms
  14. Termination condition
  15. Performance and Data Profiles
  16. ...and 5 more sections

Key Result

Lemma 3.1

Let Assumption as:accgrad hold, and suppose that at the $k$th iteration of Algorithm alg:algo where $\textcolor{black}{\theta_{p}}$ is defined in eq:baralpha. Then, and

Figures (11)

  • Figure 1: Sensitivity of the performance of restarted nonlinear conjugate gradient (NLCGr) in absence of noise (Data profiles).
  • Figure 2: Sensitivity of the performance of restarted LBFGS (LBFGSr) in absence of noise (Data profiles).
  • Figure 3: Non-restarted vs. best restarted methods in the absence of noise.
  • Figure 4: Sensitivity of the performance of restarted nonlinear conjugate gradient (NLCGr) with $\epsilon_{f}=10^{-8}$ (Data profiles).
  • Figure 5: Sensitivity of the performance of restarted LBFGS (LBFGSr) with $\epsilon_{f}=10^{-8}$ (Data profiles).
  • ...and 6 more figures

Theorems & Definitions (6)

  • Lemma 3.1
  • Lemma 3.2
  • Lemma 3.3
  • Theorem 3.1
  • Corollary 3.1
  • Remark 3.1