Bayesian Additive Regression Tree Copula Processes for Scalable Distributional Prediction

TL;DR

The paper introduces CP-BART, a distributional regression framework that constructs a Gaussian copula process from a BART model with a leaf-variance prior to capture covariate-dependent dependence in the response distribution. By representing the copula as a transport map, CP-BART enables closed-form predictive distributions and scalable Bayesian inference via MCMC, while accommodating arbitrary marginals for the response. Theoretical contributions establish posterior consistency for the pseudo-response model and convergence of predictive distributions and conditional means under mild conditions, with practical validation through simulations and five large real-world datasets showing improved density forecasts and well-calibrated predictive intervals. The work advances distributional regression with a robust, scalable approach that outperforms several benchmarks in density, quantile, and uncertainty quantification across diverse data-generating processes and applications.

Abstract

We show how to construct the implied copula process of response values from a Bayesian additive regression tree (BART) model with prior on the leaf node variances. This copula process, defined on the covariate space, can be paired with any marginal distribution for the dependent variable to construct a flexible distributional BART model. Bayesian inference is performed via Markov chain Monte Carlo on an augmented posterior, where we show that key sampling steps can be realized as those of Chipman et al. (2010), preserving scalability and computational efficiency even though the copula process is high dimensional. The posterior predictive distribution from the copula process model is derived in closed form as the push-forward of the posterior predictive distribution of the underlying BART model with an optimal transport map. Under suitable conditions, we establish posterior consistency for the regression function and posterior means and prove convergence in distribution of the predictive process and conditional expectation. Simulation studies demonstrate improved accuracy of distributional predictions compared to the original BART model and leading benchmarks. Applications to five real datasets with 506 to 515,345 observations and 8 to 90 covariates further highlight the efficacy and scalability of our proposed BART copula process model.

Figures (3)

• Figure 1: Predictive densities from CP-BART for the Rain (lefthand panel) and Bike (righthand panel) datasets. For each dataset, predictions are made for the three observations corresponding the 5th, 50th and 95th percentiles of the response variable. Solid lines depict the posterior median of the predictive densities, while the shaded areas give the 90% posterior predictive probability intervals.
• Figure 2: Pinball loss functions $\text{PL}(\alpha)$ plotted against quantile $\alpha$ for the five real-world datasets. The loss functions for the first four datasets are computed using 10-fold cross validation, and for the single pre-specified split for Music, as outlined in the text. Lower values correspond to increased accuracy.
• Figure 3: The plot shows the out-of-sample predicted conditional distributions across all methods for four different observations in the Housing dataset. The models were trained on all remaining data. The dashed vertical lines mark the observations.

Theorems & Definitions (7)

• Remark 1
• Proposition 1
• Theorem 1
• Remark 2
• Theorem 2
• Theorem 3
• Remark 3