Transforming Conditional Density Estimation Into a Single Nonparametric Regression Task

TL;DR

This work addresses conditional density estimation in high dimensions by transforming the problem into a single nonparametric regression task using auxiliary samples from an approximate identity. The proposed method, condensité, enables leveraging modern regression models (neural networks, boosted trees) to estimate the entire conditional density $f_{Y|X}$ jointly for all data points, while providing a convergence guarantee as the kernel width $h$ tends to zero. The authors establish an excess-risk bound under entropy conditions and demonstrate good out-of-the-box performance on synthetic data, a large population survey, and satellite imagery, often matching or surpassing state-of-the-art methods. The approach offers a flexible, scalable framework for regression-based conditional density estimation with strong empirical and theoretical support, broadening applicability in economics, causal inference, and ML-driven forecasting.

Abstract

We propose a way of transforming the problem of conditional density estimation into a single nonparametric regression task via the introduction of auxiliary samples. This allows leveraging regression methods that work well in high dimensions, such as neural networks and decision trees. Our main theoretical result characterizes and establishes the convergence of our estimator to the true conditional density in the data limit. We develop condensité, a method that implements this approach. We demonstrate the benefit of the auxiliary samples on synthetic data and showcase that condensité can achieve good out-of-the-box results. We evaluate our method on a large population survey dataset and on a satellite imaging dataset. In both cases, we find that condensité matches or outperforms the state of the art and yields conditional densities in line with established findings in the literature on each dataset. Our contribution opens up new possibilities for regression-based conditional density estimation and the empirical results indicate strong promise for applied research.

Figures (2)

• Figure 1: Illustration of the key idea.
• Figure 2: Illustration of the transformation of the conditional density estimation problem into a regression task. We use an approximate identity with bandwidth $h$ to define the regression task, choose a function class $\mathcal{F}_{h}$ for the regression, and optimize the fit on a set of observations.

Theorems & Definitions (1)

• Theorem 1: Excess risk bound