Fast and Accurate Bayesian Optimization with Pre-trained Transformers for Constrained Engineering Problems

TL;DR

This work introduces PFN4sBO, a fast, constraint-aware Bayesian optimization framework that uses Prior-data Fitted Networks to bypass iterative GP fitting. By implementing PFN-Pen, PFN-CEI, and PFN-CEI+ for constraint handling, the approach achieves substantial speedups (about an order of magnitude) and enhanced feasibility on a 15-problem engineering-design benchmark, outperforming traditional GP-based CEI methods. The authors provide an open-source benchmark and demonstrate that PFN-based BO, particularly PFN-CEI, delivers strong optimization performance with substantial anytime advantages, indicating significant practical potential for engineering design, interactive optimization, and multi-objective extensions. The work highlights the promise of transformer-based surrogates in constrained optimization and lays the groundwork for scalable, fast, and interactive design optimization workflows in engineering contexts.

Abstract

Bayesian Optimization (BO) is a foundational strategy in the field of engineering design optimization for efficiently handling black-box functions with many constraints and expensive evaluations. This paper introduces a fast and accurate BO framework that leverages Pre-trained Transformers for Bayesian Optimization (PFN4sBO) to address constrained optimization problems in engineering. Unlike traditional BO methods that rely heavily on Gaussian Processes (GPs), our approach utilizes Prior-data Fitted Networks (PFNs), a type of pre-trained transformer, to infer constraints and optimal solutions without requiring any iterative retraining. We demonstrate the effectiveness of PFN-based BO through a comprehensive benchmark consisting of fifteen test problems, encompassing synthetic, structural, and engineering design challenges. Our findings reveal that PFN-based BO significantly outperforms Constrained Expected Improvement and Penalty-based GP methods by an order of magnitude in speed while also outperforming them in accuracy in identifying feasible, optimal solutions. This work showcases the potential of integrating machine learning with optimization techniques in solving complex engineering challenges, heralding a significant leap forward for optimization methodologies, opening up the path to using PFN-based BO to solve other challenging problems, such as enabling user-guided interactive BO, adaptive experiment design, or multi-objective design optimization. Additionally, we establish a benchmark for evaluating BO algorithms in engineering design, offering a robust platform for future research and development in the field. This benchmark framework for evaluating new BO algorithms in engineering design will be published at https://github.com/rosenyu304/BOEngineeringBenchmark.

Figures (6)

• Figure 1: (a) GP-Pen; (b)GP-CEI/CEI+: Given an objective and $G$ constraints, GP-CEI will need $G+1$ GPs to perform one search iteration for BO. Each GP will be fit and updated in every iteration; (c) PFN-Pen; (d) PFN-CEI/CEI+: Only one PFN is needed for optimizing an objective and $G$ constraints, and no fitting of PFN will happen during BO since it is a pre-trained model. PFN's transformer nature allows the EI of the objective and $P_{feas}$ of the constraints to be solved in parallel in one pass.
• Figure 2: Overview of the details of the 15 benchmark problems. The non-feasible regions are shaded in the numerical problems. The Ackley problem is experimented with optimization in 2D, 6D, and 10D, resulting in 17 experimental trials.
• Figure 3: Box plots comparing the optimal value of each method for each experiment.
• Figure 4: Convergence plots comparing the optimal value of each method for each experiment at a fixed runtime budget. The runtime budget is set to be when PFN-Pen, the fastest method, finishes running 200 iterations. The minimum value of all methods at this fixed time budget is sorted and shown in each plot, with the value at the top being the best performance.
• Figure 5: Pareto plots demonstrating the trade-off between performance and total execution time (log-scale) for each method and test problem. D is the objective dimension, and G is the number of constraints. The average Pareto rank of each method over seventeen experiment trials is [GP-Pen, GP-CEI, GP-CEI+, PFN-Pen, PFN-CEI, PFN-CEI+] = [2.118, 2.353, 2.353, 1, 1.353, 1.765], where the smaller rank, the better, and rank 1 is the best. In problems with more than one constraint, PFN-based methods are 10 times faster than the GP-based CEI methods.