Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

QuIP: Experimental design for expensive simulators with many Qualitative factors via Integer Programming

Yen-Chun Liu, Simon Mak

TL;DR

QuIP tackles the problem of efficiently designing experiments for expensive simulators with many qualitative factors by marrying Gaussian process surrogates with an exchangeable kernel to an integer-programming framework. It reformulates both initial (maximin) and sequential (ALM and UCB) design criteria as assignment problems, enabling global optimization via state-of-the-art IP solvers like Gurobi and providing dual-optimality gaps to monitor progress. The method is validated on path-planning style problems and rover trajectory optimization, showing superior performance over metaheuristics and common baselines, especially in high-dimensional qualitative spaces. The results suggest QuIP's potential to dramatically reduce simulation budget while delivering high-quality designs, with broad applicability to discrete decision-making problems in engineering and beyond.

Abstract

The need to explore and/or optimize expensive simulators with many qualitative factors arises in broad scientific and engineering problems. Our motivating application lies in path planning - the exploration of feasible paths for navigation, which plays an important role in robotics, surgical planning and assembly planning. Here, the feasibility of a path is evaluated via expensive virtual experiments, and its parameter space is typically discrete and high-dimensional. A carefully selected experimental design is thus essential for timely decision-making. We propose here a novel framework, called QuIP, for experimental design of Qualitative factors via Integer Programming under a Gaussian process surrogate model with an exchangeable covariance function. For initial design, we show that its asymptotic D-optimal design can be formulated as a variant of the well-known assignment problem in operations research, which can be efficiently solved to global optimality using state-of-the-art integer programming solvers. For sequential design (specifically, for active learning or black-box optimization), we show that its design criterion can similarly be formulated as an assignment problem, thus enabling efficient and reliable optimization with existing solvers. We then demonstrate the effectiveness of QuIP over existing methods in a suite of path planning experiments and an application to rover trajectory optimization.

QuIP: Experimental design for expensive simulators with many Qualitative factors via Integer Programming

TL;DR

QuIP tackles the problem of efficiently designing experiments for expensive simulators with many qualitative factors by marrying Gaussian process surrogates with an exchangeable kernel to an integer-programming framework. It reformulates both initial (maximin) and sequential (ALM and UCB) design criteria as assignment problems, enabling global optimization via state-of-the-art IP solvers like Gurobi and providing dual-optimality gaps to monitor progress. The method is validated on path-planning style problems and rover trajectory optimization, showing superior performance over metaheuristics and common baselines, especially in high-dimensional qualitative spaces. The results suggest QuIP's potential to dramatically reduce simulation budget while delivering high-quality designs, with broad applicability to discrete decision-making problems in engineering and beyond.

Abstract

The need to explore and/or optimize expensive simulators with many qualitative factors arises in broad scientific and engineering problems. Our motivating application lies in path planning - the exploration of feasible paths for navigation, which plays an important role in robotics, surgical planning and assembly planning. Here, the feasibility of a path is evaluated via expensive virtual experiments, and its parameter space is typically discrete and high-dimensional. A carefully selected experimental design is thus essential for timely decision-making. We propose here a novel framework, called QuIP, for experimental design of Qualitative factors via Integer Programming under a Gaussian process surrogate model with an exchangeable covariance function. For initial design, we show that its asymptotic D-optimal design can be formulated as a variant of the well-known assignment problem in operations research, which can be efficiently solved to global optimality using state-of-the-art integer programming solvers. For sequential design (specifically, for active learning or black-box optimization), we show that its design criterion can similarly be formulated as an assignment problem, thus enabling efficient and reliable optimization with existing solvers. We then demonstrate the effectiveness of QuIP over existing methods in a suite of path planning experiments and an application to rover trajectory optimization.
Paper Structure (29 sections, 5 theorems, 28 equations, 6 figures, 2 algorithms)

This paper contains 29 sections, 5 theorems, 28 equations, 6 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background & Motivation
  3. GP modeling with qualitative variables
  4. Existing design methods
  5. QuIP: Initial Design Optimization
  6. Maximin design criterion
  7. Formulation as an assignment problem
  8. Optimization via iterative feasibility programs
  9. Algorithm statement
  10. QuIP: Sequential Design Optimization
  11. Sequential design for active learning
  12. Sequential design for black-box optimization
  13. Algorithm statement
  14. Numerical Experiments
  15. Initial design
  16. ...and 14 more sections

Key Result

Proposition 1

[Asymptotic D-optimality] Let $\mathcal{D}_{n,\boldsymbol{\theta},k}^*$ be a D-optimal design under the scaled kernel $\gamma_{{\rm E},\boldsymbol{\theta}}^k$, i.e., it maximizes $\textup{det}\{\boldsymbol{\Gamma}_{\boldsymbol{\theta},k}(\mathcal{D}_n)\}$, where $\boldsymbol{\Gamma}_{\boldsymbol{\th Here, $\zeta_k = o_k(1)$ if and only if, for any $\epsilon > 0$, $|\zeta_k| \leq \epsilon$ for $k$

Figures (6)

  • Figure 1: Plots of the maximin criterion in \ref{['eq:maximin']} (scaled by $d$) vs. computation time for different initial design optimization methods, with shaded regions showing its 95% intervals. Here, the run size is fixed at $n=50$, with a varying number of factors levels $M$ and number of factors $d$. The horizontal dotted red line shows the upper bound discussed in Section \ref{['sec:opt']} (scaled by $d$).
  • Figure 2: Visualizing the maze-solving problem. [Left] The considered $6\times 6$ maze. The red dot and yellow star mark the start and desired end point of a path, respectively, with black squares marking obstacles that cannot be passed. [Middle] Each cell shows the cost of a path that ends on such a cell. [Right] An optimal path with minimal cost.
  • Figure 3: [Left] A plot of RRMSE vs. sample size, comparing various sequential design methods for active learning of the maze-solving problem. Here, shaded regions show its 95% intervals. [Right] A plot of the current-best path cost (smaller-the-better) vs. sample size, comparing various sequential design methods for black-box optimization of the maze-solving problem.
  • Figure 4: Visualizing the $8\times 8$ greedy-snake problem, where the red dot and yellow stars mark the starting point and prizes, respectively. [Left] Visualizing the discounting reward mechanism: the reward from the green path is greater than that from the blue path as it is received earlier. [Middle] Visualizing the bonus and penalty mechanisms: the path receives a bonus at step 4 due to consecutive prizes, and a penalty at step 6 due to consecutive non-rewards. [Right] Visualizing the out-of-bounds penalty mechanism.
  • Figure 5: [Left] A plot of the current-best path reward (larger-the-better) vs. sample size, comparing various sequential design methods for black-box optimization of the greedy-snake problem. Here, shaded regions show its 95% intervals. [Right] A scatterplot of the Gurobi-estimated global optima of \ref{['eq:quipsequcb']} vs. its true oracle global optimum, for the 10-th sequential point in the greedy-snake problem. Here, each dot is for a single simulation replication, and the red line marks the ideal case where such estimates equal the oracle.
  • ...and 1 more figures

Theorems & Definitions (5)

  • Proposition 1
  • Proposition 2
  • Proposition 3
  • Proposition 4
  • Proposition 5