Global and Preference-based Optimization with Mixed Variables using Piecewise Affine Surrogates

TL;DR

This work tackles the challenge of global optimization for expensive black-box objectives over mixed-variable domains under linear QUAK constraints. It introduces PWAS, which learns a piecewise affine surrogate over a polyhedral partition and uses a MILP-based acquisition to select feasible, informative samples, with two exploration schemes to balance exploration and exploitation; a companion variant PWASp leverages pairwise preferences when objective values are unavailable. The approach includes two treatment paths for integers (categorical encoding or scaling) and a mixed-integer linear encoding of the surrogate via region indicators, enabling efficient MILP optimization. Empirical results on diverse unconstrained and constrained benchmarks show PWAS and PWASp achieve better or comparable performance with relatively few evaluations, highlighting practical utility for engineering design and related domains. The methods offer a scalable, open-source framework for mixed-variable optimization that can be extended to nonlinear constraints and integrated with other surrogate models.

Abstract

Optimization problems involving mixed variables (i.e., variables of numerical and categorical nature) can be challenging to solve, especially in the presence of mixed-variable constraints. Moreover, when the objective function is the result of a complicated simulation or experiment, it may be expensive-to-evaluate. This paper proposes a novel surrogate-based global optimization algorithm to solve linearly constrained mixed-variable problems up to medium size (around 100 variables after encoding). The proposed approach is based on constructing a piecewise affine surrogate of the objective function over feasible samples. We assume the objective function is black-box and expensive-to-evaluate, while the linear constraints are quantifiable, unrelaxable, a priori known, and are cheap to evaluate. We introduce two types of exploration functions to efficiently search the feasible domain via mixed-integer linear programming solvers. We also provide a preference-based version of the algorithm designed for situations where only pairwise comparisons between samples can be acquired, while the underlying objective function to minimize remains unquantified. The two algorithms are evaluated on several unconstrained and constrained mixed-variable benchmark problems. The results show that, within a small number of required experiments/simulations, the proposed algorithms can often achieve better or comparable results than other existing methods.

Figures (12)

• Figure 1: General procedures for surrogate-based optimization methods.
• Figure 2: The polyhedral partition induced by \ref{['eq:pwa_partition']} with $K = 10$ initial partitions. Note that the final partition is also 10 (no partition is discarded). The dots in the figure are the training data for the PARC algorithm.Different colors indicate samples at different partitions. Brownish dots next to the partition numbers are the centroid of the training data within each partition.
• Figure 3: (a) Branin function - analytical (b) Branin function fitted by PARC with $K = 10$. The dots in the figure are the test data (200 samples) for the PARC algorithm. Different colors in (b) indicate samples at different partitions.
• Figure 4: The polyhedral partition induced by \ref{['eq:pwa_partition']} with $K = 10$ initial partitions. Note that the final partition is 8 (2 partitions are discarded). The dots in the figure are the training data for the PARC algorithm.Different colors indicate samples at different partitions. Brownish dots next to the partition numbers are the centroid of the training data within each partition.
• Figure 5: (a) Function \ref{['eq:syn_fun_illustrate']} evaluated analytically, and (b) Function \ref{['eq:syn_fun_illustrate']} evaluated by the surrogates fitted by PARC with $K = 8$ final partitions.