Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Treeffuser: Probabilistic Predictions via Conditional Diffusions with Gradient-Boosted Trees

Nicolas Beltran-Velez, Alessandro Antonio Grande, Achille Nazaret, Alp Kucukelbir, David Blei

TL;DR

Treeffuser is proposed, an easy-to-use method for probabilistic prediction on tabular data that learns well-calibrated predictive distributions and can handle a wide range of regression tasks -- including those with multivariate, multimodal, and skewed responses.

Abstract

Probabilistic prediction aims to compute predictive distributions rather than single point predictions. These distributions enable practitioners to quantify uncertainty, compute risk, and detect outliers. However, most probabilistic methods assume parametric responses, such as Gaussian or Poisson distributions. When these assumptions fail, such models lead to bad predictions and poorly calibrated uncertainty. In this paper, we propose Treeffuser, an easy-to-use method for probabilistic prediction on tabular data. The idea is to learn a conditional diffusion model where the score function is estimated using gradient-boosted trees. The conditional diffusion model makes Treeffuser flexible and non-parametric, while the gradient-boosted trees make it robust and easy to train on CPUs. Treeffuser learns well-calibrated predictive distributions and can handle a wide range of regression tasks -- including those with multivariate, multimodal, and skewed responses. We study Treeffuser on synthetic and real data and show that it outperforms existing methods, providing better calibrated probabilistic predictions. We further demonstrate its versatility with an application to inventory allocation under uncertainty using sales data from Walmart. We implement Treeffuser in https://github.com/blei-lab/treeffuser.

Treeffuser: Probabilistic Predictions via Conditional Diffusions with Gradient-Boosted Trees

TL;DR

Treeffuser is proposed, an easy-to-use method for probabilistic prediction on tabular data that learns well-calibrated predictive distributions and can handle a wide range of regression tasks -- including those with multivariate, multimodal, and skewed responses.

Abstract

Probabilistic prediction aims to compute predictive distributions rather than single point predictions. These distributions enable practitioners to quantify uncertainty, compute risk, and detect outliers. However, most probabilistic methods assume parametric responses, such as Gaussian or Poisson distributions. When these assumptions fail, such models lead to bad predictions and poorly calibrated uncertainty. In this paper, we propose Treeffuser, an easy-to-use method for probabilistic prediction on tabular data. The idea is to learn a conditional diffusion model where the score function is estimated using gradient-boosted trees. The conditional diffusion model makes Treeffuser flexible and non-parametric, while the gradient-boosted trees make it robust and easy to train on CPUs. Treeffuser learns well-calibrated predictive distributions and can handle a wide range of regression tasks -- including those with multivariate, multimodal, and skewed responses. We study Treeffuser on synthetic and real data and show that it outperforms existing methods, providing better calibrated probabilistic predictions. We further demonstrate its versatility with an application to inventory allocation under uncertainty using sales data from Walmart. We implement Treeffuser in https://github.com/blei-lab/treeffuser.
Paper Structure (47 sections, 4 theorems, 30 equations, 7 figures, 8 tables, 2 algorithms)

This paper contains 47 sections, 4 theorems, 30 equations, 7 figures, 8 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Diffusion Models
  4. Gradient Boosted Trees
  5. Probabilistic Prediction via Treeffuser
  6. The Conditional Diffusion Model
  7. Conditional diffusion models.
  8. Conditional score estimation objective.
  9. The Trees
  10. Sampling and Probabilistic Predictions
  11. Limitations
  12. Related Work
  13. Diffusion models.
  14. Probabilistic prediction for tabular data
  15. Empirical studies
  16. ...and 32 more sections

Key Result

Theorem 1

Define $S^*$ as the solution of eq:conditional-objective. Then, for almost all $\boldsymbol{x}, \boldsymbol{y}, t$ with respect to $\pi(\boldsymbol{x}, \boldsymbol{y})$ and the Lebesgue measure on $t\in [0,T]$, we have

Figures (7)

  • Figure 1: Samples $\boldsymbol{y} \mid \boldsymbol{x}$ from Treeffuser vs. true densities, for multiple values of $\boldsymbol{x}$ under three different scenarios. Treeffuser captures arbitrarily complex conditional distributions that vary with $\boldsymbol{x}$.
  • Figure 2: Posterior predictive checks for Treeffuser, NGBoost Poisson, and quantile regression. Red dashed line shows the realized value on the test set. Treeffuser best captures the inflation point at zero and performs well on the tails.
  • Figure 3: Cumulative profits by method on an inventory management problem. Treeffuser produces more accurate probabilistic predictions yielding higher profits.
  • Figure 4: Visualization of ground-truth samples for the one-dimensional synthetic datasets used in the empirical studies.
  • Figure 5: Dataset size vs training time on subsets of the M5 dataset. Error bars are computed over 5 runs. Treeffuser training speed grows linearly with the size of the training points.
  • ...and 2 more figures

Theorems & Definitions (6)

  • Theorem 1: Optimal Conditional Objective
  • Theorem 2: Treeffuser One-Dimensional Objectives
  • Theorem 2: Optimal Conditional Objective
  • proof
  • Theorem 2: Treeffuser One-Dimensional Objectives
  • proof