LAGO: A Local-Global Optimization Framework Combining Trust Region Methods and Bayesian Optimization

TL;DR

LAGO, a LocAl-Global Optimization algorithm that combines gradient-enhanced Bayesian Optimization with gradient-based trust region local refinement through an adaptive competition mechanism, achieves an improved exploration of the full design space compared to standard non-linear local optimization algorithms for smooth functions, while maintaining fast local convergence in regions of interest.

Abstract

We introduce LAGO, a LocAl-Global Optimization algorithm that combines gradient-enhanced Bayesian Optimization (BO) with gradient-based trust region local refinement through an adaptive competition mechanism. At each iteration, global and local optimization strategies independently propose candidate points, and the next evaluation is selected based on predicted improvement. LAGO separates global exploration from local refinement at the proposal level: the BO acquisition function is optimized outside the active trust region, while local function and gradient evaluations are incorporated into the global gradient-enhanced Gaussian process only when they satisfy a lengthscale-based minimum-distance criterion, reducing the risk of numerical instability during the local exploitation. This enables efficient local refinement when reaching promising regions, without sacrificing a global search of the design space. As a result, the method achieves an improved exploration of the full design space compared to standard non-linear local optimization algorithms for smooth functions, while maintaining fast local convergence in regions of interest.

Figures (8)

• Figure 1: One LAGO iteration. The trust region is centered at the current best minimizer. A local candidate is proposed by the SR1 trust region step inside the region, while a global candidate is proposed by maximizing EI outside the trust region. Here, the local step will be accepted using \ref{['eq:global_condition']}. Shaded contours show the quadratic surrogate within the region and the GP posterior mean, while the black contours are the true objective values.
• Figure 2: Synthetic benchmarks. Comparison of optimization routines over synthetic test functions (Median $\pm$ IQR as shaded areas). LAGO shows an initial global exploration phase with efficient local refinement once promising regions are identified, and performs consistently well across benchmarks. Local methods such as L-BFGS and LABCAT perform well on convex or near-convex problems, but degrade on multimodal objectives such as Styblinski–Tang, while LAGO remains strong. LAGO performs similarly to BLOSSOM, but outperforms it in higher-dimensions. For gradient-based methods, each iteration is charged a cost equivalent to $d+1$ function evaluations; dashed lines indicate the median error in the idealized case in which gradient evaluations have unit cost.
• Figure 3: $\boldsymbol{\gamma}$-ablation. Performance of LAGO on the Levy function for different values of $\gamma$ in \ref{['eq:global_condition']} (default $\gamma=1$). Larger values of $\gamma$, which place greater emphasis on local refinement, lead to lower error in this synthetic benchmark.
• Figure 4: PDE-constrained optimization. Comparison of optimization techniques (Median $\pm$ IQR in shaded areas). By combining global search with gradient-informed local refinement, LAGO achieves excellent convergence in this setting.
• Figure 5: Event-aligned kernel conditioning on Branin (median and IQR (shaded areas) over 50 seeds). We align the iteration count at the first activation $t^*$ of the local assimilation filter and plot the normalized condition number $\kappa(\tilde{K}_{\nabla, t})/\kappa(\tilde{K}_{\nabla, t^*})$ within a $\pm 15$ function evaluations window. Disabling the filter ($\nu=0$) yields worse conditioning, supporting the role of selective assimilation in improving numerical stability during local trust region refinement.