CRPS-Optimal Binning for Conformal Regression

Abstract

We propose a method for non-parametric conditional distribution estimation based on partitioning covariate-sorted observations into contiguous bins and using the within-bin empirical CDF as the predictive distribution. Bin boundaries are chosen to minimise the total leave-one-out Continuous Ranked Probability Score (LOO-CRPS), which admits a closed-form cost function with $O(n^2 \log n)$ precomputation and $O(n^2)$ storage; the globally optimal $K$-partition is recovered by a dynamic programme in $O(n^2 K)$ time. Minimisation of Within-sample LOO-CRPS turns out to be inappropriate for selecting $K$ as it results in in-sample optimism. So we instead select $K$ by evaluating test CRPS on an alternating held-out split, which yields a U-shaped criterion with a well-defined minimum. Having selected $K^*$ and fitted the full-data partition, we form two complementary predictive objects: the Venn prediction band and a conformal prediction set based on CRPS as the nonconformity score, which carries a finite-sample marginal coverage guarantee at any prescribed level $\varepsilon$. On real benchmarks against split-conformal competitors (Gaussian split conformal, CQR, and CQR-QRF), the method produces substantially narrower prediction intervals while maintaining near-nominal coverage.

Figures (12)

• Figure 1: Geometric interpretation of $\mathrm{CRPS}(\hat{F}_m, y)$ as the integral of the squared vertical gap between the predictive CDF $\hat{F}_m$ (blue step function, $m = 4$ atoms) and the step $\mathbf{1}[t \ge y]$ (grey) at the observed outcome $y$. Blue shading marks intervals where $\hat{F}_m(t) > \mathbf{1}[t \ge y]$ (too little forecast mass above $t$); orange marks intervals where $\hat{F}_m(t) < \mathbf{1}[t \ge y]$ (too little mass below $t$). CRPS $= \int (\hat{F}_m(t) - \mathbf{1}[t\ge y])^2\,\mathrm{d}t$ is large when the forecast is mis-centred or over-dispersed relative to $y$.
• Figure 2: Within-sample LOO-CRPS (left) and cross-validated test CRPS (right) as functions of $K$ on the heteroscedastic example of Section \ref{['sec:example']}. The within-sample criterion is nearly monotone decreasing, confirming its susceptibility to in-sample optimism. The test CRPS has a clear U-shape with minimum at $K^*=5$. Note that the left and right $y$-axis scales differ: the left shows a total (sum over all observations), the right an average (per held-out test point).
• Figure 3: Venn prediction band (shaded) and training ECDF $\hat{F}_m$ (step function) for each of the five bins on the heteroscedastic example, alongside the true conditional CDF at the bin midpoint (dashed red). The band width $1/(m+1)$ ranges from $0.004$ to $0.009$ across bins, invisible at these bin sizes.
• Figure 4: Conformal p-value $p(y_h)$ as a function of the candidate response $y_h$ at three test points $x^* \in \{0.3, 1.5, 2.7\}$ for $\varepsilon = 0.10$. The shaded region is the prediction set $\Gamma^{0.10}$; vertical red lines mark the true $90\%$ interval under $\mathcal{N}(3x^*,(1+x^*)^2)$. The p-value curve is convex and piecewise linear, with a single connected prediction set in each case.
• Figure 5: Conformal $90\%$ prediction intervals (shaded blue) and true $90\%$ intervals (dashed red) across the covariate range, with bin boundaries marked by dotted vertical lines. The interval width adapts to the within-bin spread, tracking the increasing conditional variance of the heteroscedastic data-generating process.

Theorems & Definitions (5)

• Proposition 1
• proof
• Proposition 2
• proof
• Proposition 3: Optimal substructure