Asymmetric conformal prediction with penalized kernel sum-of-squares

TL;DR

This work addresses the gap in conformal prediction (CP) where noise can be skewed or model-bias-induced, by introducing an asymmetric CP framework based on RKHS kernel sum-of-squares (kSoS). It learns two nonnegative functions $f_{\mathrm{low}}$ and $f_{\mathrm{up}}$ to form an asymmetric score $S(X,Y)=\max\big(m_n(X)-f_{\mathrm{low}}(X)-Y,\;Y-m_n(X)-f_{\mathrm{up}}(X)\big)$, with representer theorems reducing the problem to finite PSD matrices and dual formulations enabling scalable optimization. Two symmetric penalties bridge asymmetric and symmetric bands, accompanied by a data-driven HSIC-based hyperparameter tuning strategy and warm-start optimization, promoting adaptivity in small samples or biased models. Empirical results on synthetic and real-world data show improved local coverage adaptivity while preserving marginal coverage, and demonstrate practical scalability to datasets of moderate size, with robust guidance on when asymmetry is advantageous. The approach offers a flexible, principled path to robust, adaptive CP in the presence of noise asymmetry and model bias, applicable to high-stakes regression tasks where interval accuracy and efficiency are critical.

Abstract

Conformal prediction (CP) is a distribution-free method to construct reliable prediction intervals that has gained significant attention in recent years. Despite its success and various proposed extensions, a significant practical feature which has been overlooked in previous research is the potential skewed nature of the noise, or of the residuals when the predictive model exhibits bias. In this work, we leverage recent developments in CP to propose a new asymmetric procedure that bridges the gap between skewed and non-skewed noise distributions, while still maintaining adaptivity of the prediction intervals. We introduce a new statistical learning problem to construct adaptive and asymmetric prediction bands, with a unique feature based on a penalty which promotes symmetry: when its intensity varies, the intervals smoothly change from symmetric to asymmetric ones. This learning problem is based on reproducing kernel Hilbert spaces and the recently introduced kernel sum-of-squares framework. First, we establish representer theorems to make our problem tractable in practice, and derive dual formulations which are essential for scalability to larger datasets. Second, the intensity of the penalty is chosen using a novel data-driven method which automatically identifies the symmetric nature of the noise. We show that consenting to some asymmetry can let the learned prediction bands better adapt to small sample regimes or biased predictive models.

Figures (24)

• Figure 1: Penalized kSoS with varying penalty (dataset $1$, symmetric noise). Left: asymmetric predictions bands produce tighter bands. Right: symmetric prediction bands tend to be overly conservative. An intermediate penalty value in the middle achieves tighter bands without being overly conservative and is closer to the oracle (asymmetry of prediction bands magnified with orange color).
• Figure 2: $\mathrm{HSIC}$ contour plots for asymmetric noise distribution (left) and symmetric ones (middle, right). Left: lower $\lambda_{\mathrm{pen}}$ values are clearly favored by HSIC. Middle: allowing for some asymmetry produces more adaptive bands than asymmetric or symmetric ones. Right: all $\lambda_{\mathrm{pen}}$ values achieve similar adaptivity. According to the KW test, the symmetric model is preferred over the highest $\mathrm{HSIC}$ model.
• Figure 3: Mean width (left) and absolute coverage gap combined (right) for datasets $1$, $3$ and $4$ with $n=100$, $20$ repetitions.
• Figure 4: Histogram of selected $\lambda_{\mathrm{pen}}$ among $500$ repetitions. When the sample size increases from $n=100$ to $n=200$ (top row, dataset $2$), and when the predictive model changes from a Gaussian Process to the oracle (bottom row, dataset $1$), the purely asymmetric model is selected less often. Our hyperparameter tuning method favors a symmetric model when the sample size increases and when the learned predictive model is more accurate.
• Figure 5: Left: mean width of prediction intervals on the test set for twelve real-world datasets (median$\pm$sd on 10 repetitions, values within $1\%$ of the minimum in bold). Right: worst-set coverage low/up combined for three datasets.

Theorems & Definitions (53)

• Theorem 3.1: Representer theorem
• Proposition 3.2: Dual formulation
• Theorem 3.3: Representer theorems with penalty
• Proposition 3.4: Dual formulations with penalty
• Proposition 3.8: Error bounds
• Proposition 3.9
• Theorem A.2
• proof
• Lemma A.3
• proof