Panprediction: Optimal Predictions for Any Downstream Task and Loss
Sivaraman Balakrishnan, Nika Haghtalab, Daniel Hsu, Brian Lee, Eric Zhao
TL;DR
Panprediction presents a unified framework for constructing a single probabilistic predictor that downstream users can post-process to perform well across a broad family of losses and subgroups. The key idea is a reduction to step calibration, enabling near-lossless guarantees with sample complexities that scale polylogarithmically in the number of losses, groups, and hypothesis classes. The authors provide deterministic and randomized algorithms via multi-objective learning, achieving $\\tilde{O}(\\varepsilon^{-3})$ and $\\tilde{O}(\\varepsilon^{-2})$ sample complexities respectively (up to log factors), thereby matching or surpassing existing omniprediction and multi-group learning results. By generalizing both prior paradigms, panprediction offers a principled, efficient path to joint adaptability to many losses and tasks, with potential broad impact on risk-aware decision-making across domains.
Abstract
Supervised learning is classically formulated as training a model to minimize a fixed loss function over a fixed distribution, or task. However, an emerging paradigm instead views model training as extracting enough information from data so that the model can be used to minimize many losses on many downstream tasks. We formalize a mathematical framework for this paradigm, which we call panprediction, and study its statistical complexity. Formally, panprediction generalizes omniprediction and sits upstream from multi-group learning, which respectively focus on predictions that generalize to many downstream losses or many downstream tasks, but not both. Concretely, we design algorithms that learn deterministic and randomized panpredictors with $\tilde{O}(1/\varepsilon^3)$ and $\tilde{O}(1/\varepsilon^2)$ samples, respectively. Our results demonstrate that under mild assumptions, simultaneously minimizing infinitely many losses on infinitely many tasks can be as statistically easy as minimizing one loss on one task. Along the way, we improve the best known sample complexity guarantee of deterministic omniprediction by a factor of $1/\varepsilon$, and match all other known sample complexity guarantees of omniprediction and multi-group learning. Our key technical ingredient is a nearly lossless reduction from panprediction to a statistically efficient notion of calibration, called step calibration.