Scalable neural network-based blackbox optimization
Pavankumar Koratikere, Leifur Leifsson
TL;DR
The paper tackles the high-dimensional, expensive blackbox optimization problem by replacing traditional Gaussian-process-based surrogates with a neural-network (NN) surrogate. It introduces SNBO, a three-stage sampling framework that decouples exploration and exploitation and avoids uncertainty estimation, using an adaptive perturbation range $r$ to control the search region. SNBO demonstrates strong sample efficiency and reduced runtimes across 10–50 dimensional analytical problems and two real-world tasks, often outperforming four strong baselines. The work highlights SNBO’s scalability and practical impact as a GP-free alternative for high-dimensional blackbox optimization, while also noting sensitivity to hyperparameters and opportunities for further enhancement (constrained optimization, multiple concurrent local searches).
Abstract
Bayesian Optimization (BO) is a widely used approach for blackbox optimization that leverages a Gaussian process (GP) model and an acquisition function to guide future sampling. While effective in low-dimensional settings, BO faces scalability challenges in high-dimensional spaces and with large number of function evaluations due to the computational complexity of GP models. In contrast, neural networks (NNs) offer better scalability and can model complex functions, which led to the development of NN-based BO approaches. However, these methods typically rely on estimating model uncertainty in NN prediction -- a process that is often computationally intensive and complex, particularly in high dimensions. To address these limitations, a novel method, called scalable neural network-based blackbox optimization (SNBO), is proposed that does not rely on model uncertainty estimation. Specifically, SNBO adds new samples using separate criteria for exploration and exploitation, while adaptively controlling the sampling region to ensure efficient optimization. SNBO is evaluated on a range of optimization problems spanning from 10 to 102 dimensions and compared against four state-of-the-art baseline algorithms. Across the majority of test problems, SNBO attains function values better than the best-performing baseline algorithm, while requiring 40-60% fewer function evaluations and reducing the runtime by at least an order of magnitude.