Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimizing Input Data Collection for Ranking and Selection

Eunhye Song, Taeho Kim

TL;DR

The paper tackles ranking-and-selection under input uncertainty by formulating a Bayesian framework across multiple data sources and introducing the Most Probable Best (MPB) as the estimator of the optimum. It develops OSAR, a sequential budget-allocation algorithm that optimizes input-data and simulation sampling via large-deviation-based rates, and extends it with Kernel Ridge Regression (OSAR$^+$) to improve finite-sample performance and handle continuous input spaces with strong consistency guarantees. The authors prove exponential convergence of the MPB’s posterior probability of optimality, characterize ε-optimal static sampling ratios, and demonstrate OSAR’s superior performance over a Bayesian optimization baseline across discrete and continuous input spaces and in a food-contamination application. They also propose continuous-parameter extensions (OSAR$^{++}$, OSAR$^{+FD}$, OSAR$^{+PS}$) with Nyström-KRR to manage computational costs while preserving asymptotic guarantees, offering a practical, theoretically-grounded approach to data-budgeted R&S under input uncertainty.

Abstract

We study a ranking and selection (R&S) problem when all solutions share common parametric Bayesian input models updated with the data collected from multiple independent data-generating sources. Our objective is to identify the best system by designing a sequential sampling algorithm that collects input and simulation data given a budget. We adopt the most probable best (MPB) as the estimator of the optimum and show that its posterior probability of optimality converges to one at an exponential rate as the sampling budget increases. Assuming that the input parameters belong to a finite set, we characterize the $ε$-optimal static sampling ratios for input and simulation data that maximize the convergence rate. Using these ratios as guidance, we propose the optimal sampling algorithm for R&S (OSAR) that achieves the $ε$-optimal ratios almost surely in the limit. We further extend OSAR by adopting the kernel ridge regression to improve the simulation output mean prediction. This not only improves OSAR's finite-sample performance, but also lets us tackle the case where the input parameters lie in a continuous space with a strong consistency guarantee for finding the optimum. We numerically demonstrate that OSAR outperforms a state-of-the-art competitor.

Optimizing Input Data Collection for Ranking and Selection

TL;DR

The paper tackles ranking-and-selection under input uncertainty by formulating a Bayesian framework across multiple data sources and introducing the Most Probable Best (MPB) as the estimator of the optimum. It develops OSAR, a sequential budget-allocation algorithm that optimizes input-data and simulation sampling via large-deviation-based rates, and extends it with Kernel Ridge Regression (OSAR) to improve finite-sample performance and handle continuous input spaces with strong consistency guarantees. The authors prove exponential convergence of the MPB’s posterior probability of optimality, characterize ε-optimal static sampling ratios, and demonstrate OSAR’s superior performance over a Bayesian optimization baseline across discrete and continuous input spaces and in a food-contamination application. They also propose continuous-parameter extensions (OSAR, OSAR, OSAR) with Nyström-KRR to manage computational costs while preserving asymptotic guarantees, offering a practical, theoretically-grounded approach to data-budgeted R&S under input uncertainty.

Abstract

We study a ranking and selection (R&S) problem when all solutions share common parametric Bayesian input models updated with the data collected from multiple independent data-generating sources. Our objective is to identify the best system by designing a sequential sampling algorithm that collects input and simulation data given a budget. We adopt the most probable best (MPB) as the estimator of the optimum and show that its posterior probability of optimality converges to one at an exponential rate as the sampling budget increases. Assuming that the input parameters belong to a finite set, we characterize the -optimal static sampling ratios for input and simulation data that maximize the convergence rate. Using these ratios as guidance, we propose the optimal sampling algorithm for R&S (OSAR) that achieves the -optimal ratios almost surely in the limit. We further extend OSAR by adopting the kernel ridge regression to improve the simulation output mean prediction. This not only improves OSAR's finite-sample performance, but also lets us tackle the case where the input parameters lie in a continuous space with a strong consistency guarantee for finding the optimum. We numerically demonstrate that OSAR outperforms a state-of-the-art competitor.

Paper Structure

This paper contains 29 sections, 17 theorems, 62 equations, 6 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Asymptotic Properties of the Most Probable Best
  4. Optimal Learning of the Most Probable Best
  5. Sequential Sampling Budget Allocation Design
  6. Improving the Finite-Sample Performance with Kernel Ridge Regression
  7. Extension to Continuous-valued Input Parameters
  8. Numerical Experiments
  9. Synthetic Examples
  10. Discrete Support
  11. Continuous Support
  12. Computational Overhead Comparison
  13. Contamination Control in the Food Supply Chain
  14. Conclusion
  15. Proofs and Discussions on the Theory in Section \ref{['sec:MPB_asymp']}
  16. ...and 14 more sections

Key Result

Proposition 1

For each $1\leq \ell \leq L$, let us consider an exponential family whose density $f^\ell_{\theta^\ell}(\BFx)$ is given as $f^\ell_{\theta^\ell}(\BFx) = h^\ell(\BFx)\exp\left(\left\langle\theta^\ell, S^\ell(\BFx)\right\rangle - A^\ell(\theta^\ell)\right)$. Further, assume $\Theta^\ell$ is compact an

Figures (6)

  • Figure 1: Panels (a) and (b) visualize the simulated and predicted adversarial sets, respectively. We indicate the optimal $\BFbeta$ by solving Programs \ref{['opt:plug_in_eps']} without and with $\mathcal{D}$.
  • Figure 2: Panels (a) and (b) visualize the mean functions $\{\eta_i(\BFtheta)\}_{1\leq i\leq 4}$ and each solution's favorable set, respectively.
  • Figure 3: The left panel displays $1-$PCS in log scale. The right panel presents input data sampling ratios $\hat{\beta}_{i,T}$ for $i=1,2$. The results are obtained by averaging over 10,000 macroruns.
  • Figure 4: The left panel displays $1-$PCS in log scale. The right panel presents input data sampling ratios $\hat{\beta}_{i,T}$ for $i=1,2$. The results are obtained by averaging over 10,000 macroruns.
  • Figure 5: An illustration of the two-stage food supply chain example. Each $\BFq_{i, j} = (c_{i, j}, p_{i, j})$ consists of two quantities $c_{i,j}$ and $p_{i, j}$, which represent the contamination rate and routing probability when a product transits from Processing center $i$ to $j$, respectively.
  • ...and 1 more figures

Theorems & Definitions (22)

  • Proposition 1
  • Theorem 1
  • Theorem 2
  • Corollary 1
  • Proposition 2
  • Theorem 3
  • Theorem 4
  • Remark 1
  • Proposition 3
  • Theorem 5
  • ...and 12 more