OPBO: Order-Preserving Bayesian Optimization
Wei Peng, Jianchen Hu, Kang Liu, Qiaozhu Zhai
TL;DR
High-dimensional black-box optimization is challenging for GP-based Bayesian optimization due to computational and modeling constraints. The paper introduces OPBO, an order-preserving surrogate framework that learns ordinal relationships between samples via a ranking-based neural network and selects a good-enough top-g set rather than the single optimum. Empirical results on synthetic benchmarks with dimensionality up to 1000 show OPBO consistently outperforms GP and regression NN baselines across multiple BO frameworks, with TuRBO(OP) achieving the best trade-off between speed and quality. The approach offers a robust, scalable alternative for expensive optimization tasks in engineering and ML, and opens avenues for applying OPBO to black-box constraints and broader applications.
Abstract
Bayesian optimization is an effective method for solving expensive black-box optimization problems. Most existing methods use Gaussian processes (GP) as the surrogate model for approximating the black-box objective function, it is well-known that it can fail in high-dimensional space (e.g., dimension over 500). We argue that the reliance of GP on precise numerical fitting is fundamentally ill-suited in high-dimensional space, where it leads to prohibitive computational complexity. In order to address this, we propose a simple order-preserving Bayesian optimization (OPBO) method, where the surrogate model preserves the order, instead of the value, of the black-box objective function. Then we can use a simple but effective OP neural network (NN) to replace GP as the surrogate model. Moreover, instead of searching for the best solution from the acquisition model, we select good-enough solutions in the ordinal set to reduce computational cost. The experimental results show that for high-dimensional (over 500) black-box optimization problems, the proposed OPBO significantly outperforms traditional BO methods based on regression NN and GP. The source code is available at https://github.com/pengwei222/OPBO.