Design Guidelines for Noise-Tolerant Optimization with Applications in Robust Design

TL;DR

It is shown that a new self-calibrated line search and noise-aware finite-difference techniques are effective even in the high noise regime.

Abstract

The development of nonlinear optimization algorithms capable of performing reliably in the presence of noise has garnered considerable attention lately. This paper advocates for strategies to create noise-tolerant nonlinear optimization algorithms by adapting classical deterministic methods. These adaptations follow certain design guidelines described here, which make use of estimates of the noise level in the problem. The application of our methodology is illustrated by the development of a line search gradient projection method, which is tested on an engineering design problem. It is shown that a new self-calibrated line search and noise-aware finite-difference techniques are effective even in the high noise regime. Numerical experiments investigate the resiliency of key algorithmic components. A convergence analysis of the line search gradient projection method establishes convergence to a neighborhood of stationarity.

Figures (11)

• Figure 1: Schematic plot for the design of horn
• Figure 1: Comparison of the gradient projection method with ( GP-LS) and without ( GP-F) a line search; the former using a relaxation $\epsilon_A= 10^{-3}$ and the latter using three values of $\alpha$. All methods use $N=100$ and a finite difference interval $h = 10^{-2}$. Left: Objective function value vs. iteration. Right: Objective function value vs. computational effort.
• Figure 1: Comparison of different sample sizes when using a sample consistent version of Algorithm GP-LS using sample sizes $10,50,100$, and different $\alpha_0$ respectively; Left: Objective function value vs. computational effort (up to $75,000$). Right: Objective function value vs. computational effort (up to $3\times 10^5$).
• Figure 2: Noisy Function. The vertical axis plots the noisy objective \ref{['precisely']} with different numbers of sample points: $N=10$ (left), $N=50$ (middle), and $N=100$ (right). The horizontal axes represent values of two of the design variables, $b_3$ and $b_4$. Different realizations of the random variable $\omega$ were employed for each evaluation of $\tilde{f}$ in the region of interest.
• Figure 2: Performance of Algorithm GP-LS with three values ($10^{-2}$, $10^{-3}$, $10^{-4}$) of the relaxation parameter $\epsilon_A$ in the line search. We also plot the performance of Algorithm GP-F with $\alpha = 10^{-2}$. Left: Objective function value vs. iteration. Right: Objective function vs. computational effort \ref{['comp_efforts_def']}.

Theorems & Definitions (12)

• Lemma 6.1: Prop. 1.1.9 Appendix B in bertsekas2015convex
• Lemma 6.2: Theorem 9.5-2 part (5) in kreyszig1991introductory
• Lemma 6.3
• Remark 6.4
• Lemma 6.5
• Proof 1
• Lemma 6.6
• Proof 2
• Theorem 6.7
• Proof 3