Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Estimating Computational Noise on Parametric Curves

Matt Menickelly

TL;DR

It is shown that in many situations, noise estimates obtained from providing an arbitrary geometry of points as input to ECNoise are often indistinguishable from noise estimates obtained from using the standard geometry, and suggests a practical means by which one can gradually adjust an initial arbitrary point selection to yield better noise estimates with higher probability.

Abstract

We consider ECNoise, a practical tool for estimating the magnitude of noise in evaluations of a black-box function. Recent developments in numerical optimization algorithms have seen increased usage of ECNoise as a subroutine to provide a solver with noise level estimates, so that the solver might somehow proportionally adjust for noise. Particularly motivated by problems in computationally expensive derivative-free optimization, we question a fundamental assumption made in the original development of ECNoise, particularly the assumption that the set of points provided to ECNoise must satisfy fairly restrictive geometric conditions (in particular, that the points be collinear and equally spaced). Driven by prior practical experience, we show that in many situations, noise estimates obtained from providing an arbitrary (that is, not collinear) geometry of points as input to ECNoise are often indistinguishable from noise estimates obtained from using the standard (collinear and equally spaced) geometry. We analyze this via parametric curves that interpolate the arbitrary input points. The analysis provides insight into the circumstances in which one can expect arbitrary point selection to cause significant degradation of ECNoise. Moreover, the analysis suggests a practical means (the solution of a small mixed integer linear program) by which one can gradually adjust an initial arbitrary point selection to yield better noise estimates with higher probability.

Estimating Computational Noise on Parametric Curves

TL;DR

It is shown that in many situations, noise estimates obtained from providing an arbitrary geometry of points as input to ECNoise are often indistinguishable from noise estimates obtained from using the standard geometry, and suggests a practical means by which one can gradually adjust an initial arbitrary point selection to yield better noise estimates with higher probability.

Abstract

We consider ECNoise, a practical tool for estimating the magnitude of noise in evaluations of a black-box function. Recent developments in numerical optimization algorithms have seen increased usage of ECNoise as a subroutine to provide a solver with noise level estimates, so that the solver might somehow proportionally adjust for noise. Particularly motivated by problems in computationally expensive derivative-free optimization, we question a fundamental assumption made in the original development of ECNoise, particularly the assumption that the set of points provided to ECNoise must satisfy fairly restrictive geometric conditions (in particular, that the points be collinear and equally spaced). Driven by prior practical experience, we show that in many situations, noise estimates obtained from providing an arbitrary (that is, not collinear) geometry of points as input to ECNoise are often indistinguishable from noise estimates obtained from using the standard (collinear and equally spaced) geometry. We analyze this via parametric curves that interpolate the arbitrary input points. The analysis provides insight into the circumstances in which one can expect arbitrary point selection to cause significant degradation of ECNoise. Moreover, the analysis suggests a practical means (the solution of a small mixed integer linear program) by which one can gradually adjust an initial arbitrary point selection to yield better noise estimates with higher probability.
Paper Structure (6 sections, 5 theorems, 35 equations, 3 figures, 1 table)

This paper contains 6 sections, 5 theorems, 35 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Introduction to ECNoise
  3. Motivation
  4. ECNoise on Parametric Curves
  5. Understanding \ref{['thm:kcd']}
  6. A Practical Method for Down-selecting Sample Points

Key Result

theorem thmcountertheorem

If $\{\xi(p(j)) : j = 0,1,\dots,m\}$ are iid realizations of the random variable $\Xi(y^0,\delta)$, then where $\gamma_k = (k!)^2/(2k)!$ for each $k=1,2,\dots,m$.

Figures (3)

  • Figure 1: Splined histograms of relative noise values returned from running the described experiment. Left figure corresponds to running ECNoise in "standard" mode; right figure corresponds to running ECNoise with arbitrary function values.
  • Figure 2: Comparing the "standard" use of ECNoise (in dashed blue) to arbitrary point selection (in solid red) on the synthetic problem with smooth part defined by \ref{['eq:synthetic']} over varying values of $h$ and $n$. These plots are empirical cumulative density functions, over 10,000 trials, of the noise level estimate returned by ECNoise. We note that ECnoise may decline to provide a noise level estimate if it detects $h$ is too large. In this case the noise level estimate is set to 0. This accounts for, in the larger values of $h$ on the left of this grid, the left-hand side of the empirical cumulative distribution function beginning at a probability value greater than 0. The limit of the $x$-axis is chosen as 4 times the correct value of $\epsilon_f = 10^{-3}$, which was the threshold specified for successful noise estimation specified in more2011estimating. We also show, in each subplot via black dotted lines, the location of $\epsilon_f=10^{-3}$ on the $x$-axis and the location of $0.5$ on the $y$-axis, since we expect the median of the noise level estimate distribution to lie around $10^{-3}$.
  • Figure 3: Repeating the experiment in \ref{['fig:compare_reuse']} corresponding to $n=6$ and $h=10^{-6}$, with the same empirical cdfs. For reference, we continue to display empirical cdfs for the "standard" use of ENnoise and the use of ECNoise with a randomly generated set of points as in the prior experiment. This time, in each trial, we also solve the MILP in \ref{['eq:milp_R']} with the specified $R$ and the identical arbitrary point set used to generate the noise estimate in the arbitrary case. The left plot shows $m=6$, while the right plot shows $m=12$.

Theorems & Definitions (9)

  • theorem thmcountertheorem
  • proof
  • theorem thmcountertheorem
  • proof
  • lemma thmcounterlemma
  • theorem thmcountertheorem
  • proof
  • theorem thmcountertheorem
  • proof