Calibrated Multi-Level Quantile Forecasting

TL;DR

This work introduces MultiQT, an online recalibration framework that wraps any base forecaster to produce calibrated, noncrossing multi-level quantile forecasts with a no-regret guarantee. By recasting calibration as constrained gradient equilibrium and employing lazy gradient descent with an isotonic projection, MultiQT ensures long-run calibration at all levels while preserving forecast sharpness. The authors prove calibration and regret guarantees under standard Lipschitz, restorativity, and inward-flow conditions, and extend to delayed feedback. Empirical results on COVID-19 death forecasting and energy production demonstrate improved calibration with little to no degradation in aggregated quantile loss, highlighting practical benefits for decision-making under uncertainty.

Abstract

We present an online method for guaranteeing calibration of quantile forecasts at multiple quantile levels simultaneously. A sequence of $α$-level quantile forecasts is calibrated if the forecasts are larger than the target value at an $α$-fraction of time steps. We introduce a lightweight method called Multi-Level Quantile Tracker (MultiQT) that wraps around any existing point or quantile forecaster to produce corrected forecasts guaranteed to achieve calibration, even against adversarial distribution shifts, while ensuring that the forecasts are ordered -- e.g., the 0.5-level quantile forecast is never larger than the 0.6-level forecast. Furthermore, the method comes with a no-regret guarantee that implies it will not worsen the performance of an existing forecaster, asymptotically, with respect to the quantile loss. In experiments, we find that MultiQT significantly improves the calibration of real forecasters in epidemic and energy forecasting problems.

Figures (22)

• Figure 1: One-week-ahead forecasts of weekly COVID-19 deaths in California from July 11, 2020 to October 22, 2022 generated by forecaster RobertWalraven-ESG before (top) and after (bottom) applying MultiQT. Forecasts are made at levels 0.01, 0.025, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, 0.975, and 0.99. To visualize these forecasts, we plot colored bands where the lightest opacity connects the 0.01 and 0.99 level forecasts, the next lightest connects the 0.025 and 0.975 level forecasts, and so on.
• Figure 2: To prove that MultiQT achieves calibration without crossings, we will first show that lazy gradient descent achieves constrained gradient equilibrium (constrained GEQ). MultiQT then inherits the desired guarantee.
• Figure 3: Visualization of inward flow. The blue arrows represent the negative gradient $-g$ evaluated at points on the boundary of the constraint set $C$.
• Figure 4: Actual coverage vs. desired coverage at each quantile level for one-week-ahead COVID-19 death forecasts before (red) and after (blue) applying MultiQT. Each forecaster $\times$ state combination is a line.
• Figure 5: Average quantile loss and average calibration error for raw forecasts (tail of arrow) and MultiQT forecasts (head of arrow) for $h$-week-ahead COVID-19 death forecasts, where $h\in\{1,2,3,4\}$. Each color represents a forecaster, and the coordinates of the head and tail are determined by averaging metrics across all 50 states for the specified horizon. For both metrics, lower is better.

Theorems & Definitions (50)

• Proposition 1: Quantile Tracker guarantee from angelopoulos2023conformal
• Proposition 2: Post hoc ordering of Quantile Tracker fails
• Definition 1: angelopoulos2025gradient
• Definition 2
• Definition 3: Restorativity, angelopoulos2025gradient
• Proposition 3: Online gradient descent achieves GEQ, Proposition 5 of angelopoulos2025gradient
• Proposition 4: Projected gradient descent fails
• Definition 4: Inward flow
• Proposition 5: Lazy gradient descent achieves GEQ under inward flow
• Proposition 6: Lazy gradient descent with delay achieves GEQ under inward flow