Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ReMU: Regional Minimal Updating for Model-Based Derivative-Free Optimization

Pengcheng Xie, Stefan M. Wild

TL;DR

This paper extends derivative-free model-based optimization by introducing regional minimal updating (ReMU) quadratic models, which blend zeroth-, first-, and second-order information through weights $C_1,C_2,C_3$ in a local region. The authors derive explicit ReMU formulations via a KKT system, analyze the KKT matrix distance and the geometry of the weight region, and show that a barycentric choice $C_1=C_2=C_3= frac{1}{3}$ often yields favorable performance. Numerical experiments on Rosenbrock, Osborne2, and a JJMSMW09 benchmark demonstrate that barycentric ReMU models, and especially corrected ReMU variants integrated into POUNDerS, deliver improved accuracy and robustness across problem classes and noise levels. The work also introduces the truncated Newton step error as a diagnostic for model quality and proposes adaptive weight-correction strategies that enhance solver performance in online settings. Overall, ReMU offers a flexible, theoretically grounded framework to enhance interpolation-based, derivative-free trust-region methods in scientific computing contexts.

Abstract

Derivative-free optimization (DFO) problems are optimization problems where derivative information is unavailable or extremely difficult to obtain. Model-based DFO solvers have been applied extensively in scientific computing. Powell's NEWUOA (2004) and Wild's POUNDerS (2014) explore the numerical power of the minimal norm Hessian (MNH) model for DFO and contributed to the open discussion on building better models with fewer data to achieve faster numerical convergence. Another decade later, we propose the regional minimal updating (ReMU) models, and extend the previous models into a broader class. This paper shows motivation behind ReMU models, computational details, theoretical and numerical results on particular extreme points and the barycenter of ReMU's weight coefficient region, and the associated KKT matrix error and distance. Novel metrics, such as the truncated Newton step error, are proposed to numerically understand the new models' properties. A new algorithmic strategy, based on iteratively adjusting the ReMU model type, is also proposed, and shows numerical advantages by combining and switching between the barycentric model and the classic least Frobenius norm model in an online fashion.

ReMU: Regional Minimal Updating for Model-Based Derivative-Free Optimization

TL;DR

This paper extends derivative-free model-based optimization by introducing regional minimal updating (ReMU) quadratic models, which blend zeroth-, first-, and second-order information through weights in a local region. The authors derive explicit ReMU formulations via a KKT system, analyze the KKT matrix distance and the geometry of the weight region, and show that a barycentric choice often yields favorable performance. Numerical experiments on Rosenbrock, Osborne2, and a JJMSMW09 benchmark demonstrate that barycentric ReMU models, and especially corrected ReMU variants integrated into POUNDerS, deliver improved accuracy and robustness across problem classes and noise levels. The work also introduces the truncated Newton step error as a diagnostic for model quality and proposes adaptive weight-correction strategies that enhance solver performance in online settings. Overall, ReMU offers a flexible, theoretically grounded framework to enhance interpolation-based, derivative-free trust-region methods in scientific computing contexts.

Abstract

Derivative-free optimization (DFO) problems are optimization problems where derivative information is unavailable or extremely difficult to obtain. Model-based DFO solvers have been applied extensively in scientific computing. Powell's NEWUOA (2004) and Wild's POUNDerS (2014) explore the numerical power of the minimal norm Hessian (MNH) model for DFO and contributed to the open discussion on building better models with fewer data to achieve faster numerical convergence. Another decade later, we propose the regional minimal updating (ReMU) models, and extend the previous models into a broader class. This paper shows motivation behind ReMU models, computational details, theoretical and numerical results on particular extreme points and the barycenter of ReMU's weight coefficient region, and the associated KKT matrix error and distance. Novel metrics, such as the truncated Newton step error, are proposed to numerically understand the new models' properties. A new algorithmic strategy, based on iteratively adjusting the ReMU model type, is also proposed, and shows numerical advantages by combining and switching between the barycentric model and the classic least Frobenius norm model in an online fashion.

Paper Structure

This paper contains 15 sections, 8 theorems, 52 equations, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. Model-based method using ReMU models
  3. A model-based algorithm using ReMU models
  4. Formula of regional minimal updating
  5. Numerical results of the algorithm with the ReMU models
  6. A numerical example
  7. Performance and date profiles for test set
  8. Truncated Newton step error comparison
  9. Regional minimal updating
  10. KKT matrix error and distance
  11. Geometric points of the coefficient region
  12. Model-based methods using the corrected ReMU models
  13. Model-based algorithm using corrected ReMU models
  14. Numerical results
  15. Conclusions

Key Result

Theorem 1.3

Given a quadratic objective function $f$, if $m_k$ is the solution of eq:weightproblem, then it satisfies

Figures (8)

  • Figure 1: Performance (top row) and data (bottom row) profiles with accuracy levels $\tau=10^{-1},10^{-2},10^{-3},10^{-4}$ (from left to right) for $2n+1$ interpolation points at each step
  • Figure 2: Performance (top row) and data (bottom row) profiles with accuracy levels $\tau=10^{-1},10^{-2},10^{-3},10^{-4}$ (from left to right) for $n+3$ interpolation points at each step
  • Figure 3: Truncated Newton step error ${\rm Dist}_{\mathcal{N}}(m_k,f,{\boldsymbol{x}}_k,\Delta_k)$ for different ReMU weight coefficients. The boxes and central line show the 30%, 50%, and 70% quantiles across 100 iterations.
  • Figure 4: Coefficient region $\mathcal{C}$
  • Figure 5: Performance (1st column) and data (2nd column) profiles with accuracy levels $\tau=10^{-1},10^{-2},10^{-3},10^{-6}$ (from top to bottom) for $|\mathcal{X}_k|=2n+1$ interpolation points at each step; for the noisy problems, $\sigma=10^{-2}$.
  • ...and 3 more figures

Theorems & Definitions (27)

  • Definition 1.1
  • Definition 1.2: Regional minimal updating quadratic model (ReMU model)
  • Theorem 1.3
  • proof
  • Lemma 2.1
  • Remark 1
  • Example 2.2
  • Definition 2.3: Truncated Newton step
  • Definition 2.4: Truncated Newton step error
  • Proposition 3.1
  • ...and 17 more