DistPred: A Distribution-Free Probabilistic Inference Method for Regression and Forecasting

TL;DR

DistPred addresses the need for uncertainty quantification in regression and forecasting by learning a distribution-free predictor that outputs an ensemble of $K$ predictive values in a single forward pass. It converts proper scoring rules into a differentiable, memory-efficient loss $C(\hat{F},y)$ based on full predictive quantiles, enabling estimation of the full predictive distribution and confidence intervals without assuming a parametric form. The approach achieves state-of-the-art performance on UCI regression tasks and multivariate time-series forecasting while offering substantial speedups (notably $\approx$180x faster inference) and flexibility to combine with methods like MC Dropout. Through extensive experiments and visualization, the paper demonstrates accurate distribution reconstruction, calibration, and practical utility for uncertainty quantification in real-world regression and forecasting settings.

Abstract

Traditional regression and prediction tasks often only provide deterministic point estimates. To estimate the distribution or uncertainty of the response variable, traditional methods either assume that the posterior distribution of samples follows a Gaussian process or require thousands of forward passes for sample generation. We propose a novel approach called DistPred for regression and forecasting tasks, which overcomes the limitations of existing methods while remaining simple and powerful. Specifically, we transform proper scoring rules that measure the discrepancy between the predicted distribution and the target distribution into a differentiable discrete form and use it as a loss function to train the model end-to-end. This allows the model to sample numerous samples in a single forward pass to estimate the potential distribution of the response variable. We have compared our method with several existing approaches on multiple datasets and achieved state-of-the-art performance. Additionally, our method significantly improves computational efficiency. For example, compared to state-of-the-art models, DistPred has a 180x faster inference speed Experimental results can be reproduced through https://github.com/Anoise/DistPred.

Figures (10)

• Figure 1: DistPred can provide $K$ predicted values $\hat{y}$ of the response variable $y$ given the predictor variable $x$ in a single forward process, denoted as $\mathbb{E}(\hat{Y}|x)$, where $\hat{Y}$ represents a maximum likelihood sample of $y$. Based on this sampling, the probability mass function (PMF) $P_{\theta}(\hat{y}|x)$, cumulative distribution function (CDF) $F_{\theta}(\hat{y}|x)$, and confidence curve $CC_{\theta}(\hat{y}|x)$ for the response variable $y$ can be computed, thereby yielding comprehensive statistical insights into $y$. For instance, this includes confidence intervals (CI) at any desired level, as well as p-values.
• Figure 2: The workflow of DistPred. An ensemble of predictive variables $\hat{Y}$ is inferred in a forward pass and $S(\mathbb{E}(\hat{Y}|x),y)$ is utilized to train the learner end-to-end.
• Figure 3: DistPred's architecture. A model $M$ with parameters $\theta$ takes one input variable $x$ and outputs an ensemble of $K$ response variable $\hat{y}_1, \cdots, \hat{y}_K$.
• Figure 4: Scatter plot of DistPred's regression results on $8$ toy examples.
• Figure 5: DistPred and DistPred+MCD on UCI datasets.