Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Simulating classification models to evaluate Predict-Then-Optimize methods

Pieter Smet

TL;DR

The paper develops and validates simulation-based methods to assess how classification prediction errors affect Predict-Then-Optimize decisions, extending prior binaryclassifier approaches to multiclass settings. Using a single-machine scheduling problem, it shows that the relationship between predictive accuracy and decision quality is nuanced and highly instance-dependent, and demonstrates that simulation can estimate performance requirements without training numerous models. The findings support using simulation to guide model selection and performance targets before deployment, while highlighting the role of error structure. The work also outlines future directions, including improved multiclass simulation and integration with decision-focused learning paradigms.

Abstract

Uncertainty in optimization is often represented as stochastic parameters in the optimization model. In Predict-Then-Optimize approaches, predictions of a machine learning model are used as values for such parameters, effectively transforming the stochastic optimization problem into a deterministic one. This two-stage framework is built on the assumption that more accurate predictions result in solutions that are closer to the actual optimal solution. However, providing evidence for this assumption in the context of complex, constrained optimization problems is challenging and often overlooked in the literature. Simulating predictions of machine learning models offers a way to (experimentally) analyze how prediction error impacts solution quality without the need to train real models. Complementing an algorithm from the literature for simulating binary classification, we introduce a new algorithm for simulating predictions of multiclass classifiers. We conduct a computational study to evaluate the performance of these algorithms, and show that classifier performance can be simulated with reasonable accuracy, although some variability is observed. Additionally, we apply these algorithms to assess the performance of a Predict-Then-Optimize algorithm for a machine scheduling problem. The experiments demonstrate that the relationship between prediction error and how close solutions are to the actual optimum is non-trivial, highlighting important considerations for the design and evaluation of decision-making systems based on machine learning predictions.

Simulating classification models to evaluate Predict-Then-Optimize methods

TL;DR

The paper develops and validates simulation-based methods to assess how classification prediction errors affect Predict-Then-Optimize decisions, extending prior binaryclassifier approaches to multiclass settings. Using a single-machine scheduling problem, it shows that the relationship between predictive accuracy and decision quality is nuanced and highly instance-dependent, and demonstrates that simulation can estimate performance requirements without training numerous models. The findings support using simulation to guide model selection and performance targets before deployment, while highlighting the role of error structure. The work also outlines future directions, including improved multiclass simulation and integration with decision-focused learning paradigms.

Abstract

Uncertainty in optimization is often represented as stochastic parameters in the optimization model. In Predict-Then-Optimize approaches, predictions of a machine learning model are used as values for such parameters, effectively transforming the stochastic optimization problem into a deterministic one. This two-stage framework is built on the assumption that more accurate predictions result in solutions that are closer to the actual optimal solution. However, providing evidence for this assumption in the context of complex, constrained optimization problems is challenging and often overlooked in the literature. Simulating predictions of machine learning models offers a way to (experimentally) analyze how prediction error impacts solution quality without the need to train real models. Complementing an algorithm from the literature for simulating binary classification, we introduce a new algorithm for simulating predictions of multiclass classifiers. We conduct a computational study to evaluate the performance of these algorithms, and show that classifier performance can be simulated with reasonable accuracy, although some variability is observed. Additionally, we apply these algorithms to assess the performance of a Predict-Then-Optimize algorithm for a machine scheduling problem. The experiments demonstrate that the relationship between prediction error and how close solutions are to the actual optimum is non-trivial, highlighting important considerations for the design and evaluation of decision-making systems based on machine learning predictions.

Paper Structure

This paper contains 13 sections, 3 equations, 7 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Evaluating PTO methods
  3. Simulating classification models
  4. Binary classification
  5. Multiclass classification
  6. Computational study
  7. Problem definition
  8. Data and experimental setup
  9. Results
  10. Approximation error
  11. Error sensitivity analysis
  12. Actual PTO methods
  13. Conclusions

Figures (7)

  • Figure 1: Schematic overview of the PTO for total weighted completion time minimization with uncertain job types.
  • Figure 2: Comparison of actual and simulated TPR and FPR for $T=2$.
  • Figure 3: Comparison of actual and simulated TPR and FPR for $T=3$.
  • Figure 4: Histogram of absolute gaps between actual and predicted TPR. The bin width is 0.005.
  • Figure 5: Gap to the actual optimal total weighted completion time for different prediction models for $T=2$ and $\eta=20$ (sseed=7). The dotted line indicates for which TPR and FRP values the gap is at most 1%, the dashed line for 5% and the solid line for 10%.
  • ...and 2 more figures