Algorithms with Calibrated Machine Learning Predictions

Abstract

The field of algorithms with predictions incorporates machine learning advice in the design of online algorithms to improve real-world performance. A central consideration is the extent to which predictions can be trusted -- while existing approaches often require users to specify an aggregate trust level, modern machine learning models can provide estimates of prediction-level uncertainty. In this paper, we propose calibration as a principled and practical tool to bridge this gap, demonstrating the benefits of calibrated advice through two case studies: the ski rental and online job scheduling problems. For ski rental, we design an algorithm that achieves near-optimal prediction-dependent performance and prove that, in high-variance settings, calibrated advice offers more effective guidance than alternative methods for uncertainty quantification. For job scheduling, we demonstrate that using a calibrated predictor leads to significant performance improvements over existing methods. Evaluations on real-world data validate our theoretical findings, highlighting the practical impact of calibration for algorithms with predictions.

Figures (9)

• Figure 1: Job sequencing under fine-grained (above) and coarse (below) calibrated predictors. For six example jobs, predicted probabilities $p_i$ are marked with $\times$, and numbered boxes give the order of jobs according to each predictor.
• Figure 2: Comparison of $\mathbb{E}[\textnormal{ALG}/\textnormal{OPT}]$ for algorithms aided by predictions from a small MLP with two hidden layers of size 8 and 2. Algorithm \ref{['alg: optimal-ski-rental']} (Calibrated) performs best on average.
• Figure 3: Comparison of $\mathop{\mathbb{E}}[\textnormal{ALG}-\textnormal{OPT}]$ (normalized) achieved by \ref{['alg: beta-threshold']} for naively calibrated and histogram-binned predictors under varying delay costs $\omega_0, \omega_1$ and information barrier $\theta$.
• Figure 4: Distribution of ride times and quantiles in minutes, most rides are under 900 minutes.
• Figure 5: Predictor accuracy with different features around final docking station, no information, partial information (approximate latitude), and rich information (approximate latitude and longitude).

Theorems & Definitions (39)

• Definition 2.1
• Definition 2.2
• Theorem 3.1
• Lemma 3.1
• proof : Proof sketch
• Theorem 3.2
• proof : Proof sketch
• Theorem 3.3
• Lemma 3.3
• proof : Proof of \ref{['thm: ski-rental-cr']}