Benchmarking PtO and PnO Methods in the Predictive Combinatorial Optimization Regime

TL;DR

A modular framework to benchmark 11 existing PtO/PnO methods on 8 problems, including a new industrial dataset for combinatorial advertising that will be released, and shows that PnO approaches are better than PtO on 7 out of 8 benchmarks, but there is no silver bullet found for the specific design choices of PnO.

Abstract

Predictive combinatorial optimization, where the parameters of combinatorial optimization (CO) are unknown at the decision-making time, is the precise modeling of many real-world applications, including energy cost-aware scheduling and budget allocation on advertising. Tackling such a problem usually involves a prediction model and a CO solver. These two modules are integrated into the predictive CO pipeline following two design principles: "Predict-then-Optimize (PtO)", which learns predictions by supervised training and subsequently solves CO using predicted coefficients, while the other, named "Predict-and-Optimize (PnO)", directly optimizes towards the ultimate decision quality and claims to yield better decisions than traditional PtO approaches. However, there lacks a systematic benchmark of both approaches, including the specific design choices at the module level, as well as an evaluation dataset that covers representative real-world scenarios. To this end, we develop a modular framework to benchmark 11 existing PtO/PnO methods on 8 problems, including a new industrial dataset for combinatorial advertising that will be released. Our study shows that PnO approaches are better than PtO on 7 out of 8 benchmarks, but there is no silver bullet found for the specific design choices of PnO. A comprehensive categorization of current approaches and integration of typical scenarios are provided under a unified benchmark. Therefore, this paper could serve as a comprehensive benchmark for future PnO approach development and also offer fast prototyping for application-focused development. The code is available at https://github.com/Thinklab-SJTU/PredictiveCO-Benchmark.

Figures (10)

• Figure 1: (a) Example of predictive CO in energy-cost aware scheduling. Factories deploy job schedules based on energy price predictions to reduce production costs. (b) Visualization of energy prices of the 200th test instance in SEMO dataset. PtO makes improper predictions, which further prescribes sub-optimal decisions. (c) PtO vs. PnO. PnO designs decision-oriented training approaches that emerged recently as a promising direction to tackle predictive CO.
• Figure 2: Gradient flow of existing PnO methods, where $f$, $\hat{f}$ and $\tilde{f}$ denote the original objective, learned objective by surrogate model, and the continuous relaxation of the original objective, respectively. The path of back propagation of the vanilla two-stage is ①, while discrete/continuous categories use ①②, statistical one uses ④, and surrogate one uses ①③. $l_{surro}$ represents the surrogate losses to measure how the surrogate imitates the original optimization objective, while $l_{rank}$ refers to the designed loss that encodes solution ranking—e.g., ensuring that the optimal solution is assigned a lower loss than suboptimal ones.
• Figure 3: A modular code framework supporting 11 problems, 8 PtO/PnO models, multiple solvers, and various evaluations under configurable parameters. Users can easily customize their own problems, predictors, models, and solvers.
• Figure 4: Choice guide of PnO models, which depends on optimization objective type, available resources. From top to bottom, it becomes harder to deploy PnO for harder optimization types and fewer available resources.
• Figure 5: (a b) Results of fine-tuning (short as "FTN") of the discrete category (BB, ID model). Compared to training PnO directly, "BB-FTN" and "ID-FTN" benefit by first pertaining by PtO and then fine-tuning by PnO on 4,5 over 7 datasets, respectively. (c d) Learning curve on knapsack (gen) dataset for prediction loss and decision regret w.r.t. training epochs. PnO approaches (LTR, SPO) achieve lower regret than the PtO approach (Two-stage), though with higher prediction loss.