Sample-Efficient Omniprediction for Proper Losses

TL;DR

The paper tackles omniprediction for multiple downstream decision-makers by seeking a single universal predictor that minimizes loss across all downstream utilities when paired with proper losses. It clarifies that unrestricted comparator classes render omniprediction as $L_1$-estimation of the true probability $p^*(X)$, suffering from the curse of dimensionality, and shows that boosting approaches aiming for multicalibration or calibrated multiaccuracy incur larger sample complexities. It then develops two sample-efficient, largely unrandomized strategies: (i) a two-player online game method achieving the optimal rate $\tilde{O}_{\mathbb{P}}\big(\sqrt{\mathrm{VC}(\mathcal{F})/n}\big)$, and (ii) a direct ensembling procedure that builds a single predictor by iteratively merging threshold-based predictors with provable guarantees at the same rate. Empirical results on simulated data and a sales forecasting task show CMA to be inferior, while both two-player and direct ensembling perform comparably well, with the latter offering practical simplicity and the former robust at small sample sizes. The work also discusses extensions to other elicitable targets and highlights open questions for multiclass settings.

Abstract

We consider the problem of constructing probabilistic predictions that lead to accurate decisions when employed by downstream users to inform actions. For a single decision maker, designing an optimal predictor is equivalent to minimizing a proper loss function corresponding to the negative utility of that individual. For multiple decision makers, our problem can be viewed as a variant of omniprediction in which the goal is to design a single predictor that simultaneously minimizes multiple losses. Existing algorithms for achieving omniprediction broadly fall into two categories: 1) boosting methods that optimize other auxiliary targets such as multicalibration and obtain omniprediction as a corollary, and 2) adversarial two-player game based approaches that estimate and respond to the ``worst-case" loss in an online fashion. We give lower bounds demonstrating that multicalibration is a strictly more difficult problem than omniprediction and thus the former approach must incur suboptimal sample complexity. For the latter approach, we discuss how these ideas can be used to obtain a sample-efficient algorithm through an online-to-batch conversion. This conversion has the downside of returning a complex, randomized predictor. We improve on this method by designing a more direct, unrandomized algorithm that exploits structural elements of the set of proper losses.

Figures (4)

• Figure 1: Illustration of the core ensembling problem for our simulated example (left panel) and realized average omniprediction error of the calibrated multiaccuracy (blue), two-player game based (orange), and direct ensembling (green) methods across various sample sizes with $m = 16$ fixed (center panel) or chosen variably as $m = 2^{\lfloor \log_2(\sqrt{n}) \rfloor}$ (right panel). Dots and error bars show means and standard errors obtained by evaluating the omniprediction error over 2000 test points for each of 40 draws of the training data. Hyperparameters for the calibrated multiaccuracy, two-player, and direct ensembling methods are set as $c = 0.5$, $c=32$, and $c=0$, respectively.
• Figure 2: Omniprediction error of the calibrated multiaccuracy (left panels), two-player game based (center panels), and direct ensembling (right panels) methods across various sample sizes with $m = 16$ fixed (top row) or chosen variably as $m = 2^{\lfloor \log_2(\sqrt{n}) \rfloor}$ (bottom row) as the scaling constant $c$ varies on a simulated dataset. Dots and error bars show means and standard errors obtained by evaluating the omniprediction error over 2000 test points for each of 40 draws of the training data.
• Figure 3: Realized average omniprediction error of the calibrated multiaccuracy (blue), two-player game based (orange), and direct ensembling (green) methods as well as the error of the best base model (red) across various sample sizes with $m=16$ fixed (left panel) or chosen variably as $m = 2^{\lfloor \log_2(\sqrt{n}) \rfloor}$ (right panel) on the M5 sales forecasting dataset. Dots and error bars show estimated means and standard errors obtained by evaluating the omniprediction error over 2000 test points for each of 20 draws of the training data. Hyperparameters for the calibrated multiaccuracy, two-player, and direct ensembling methods are set as $c = 0.5$, $c=32$, and $c=0$, respectively.
• Figure 4: Omniprediction error of the calibrated multiaccuracy (left panels), two-player game based (center panels), and direct ensembling (right panels) methods across various sample sizes with $m = 16$ fixed (top row) or chosen variably as $m = 2^{\lfloor \log_2(\sqrt{n}) \rfloor}$ (bottom row) as the scaling constant $c$ varies on the M5 sales forecasting dataset. Dots and error bars show means and standard errors obtained by evaluating the omniprediction error over 2000 test points for each of 20 draws of the training data.

Theorems & Definitions (33)

• Proposition 1
• Theorem 1: Corollary of Lemma 12, Proposition 13, and Theorem 17 in Goplan2023_OI
• Proposition 2
• Proposition 3
• Proposition 4
• Theorem 2: Theorem 1 of Ehm2016
• Lemma 1
• Lemma 2
• proof
• Lemma 3