Kernel Density Estimators in Large Dimensions

Abstract

This paper studies Kernel Density Estimation for a high-dimensional distribution $ρ(x)$. Traditional approaches have focused on the limit of large number of data points $n$ and fixed dimension $d$. We analyze instead the regime where both the number $n$ of data points $y_i$ and their dimensionality $d$ grow with a fixed ratio $α=(\log n)/d$. Our study reveals three distinct statistical regimes for the kernel-based estimate of the density $\hat ρ_h^{\mathcal {D}}(x)=\frac{1}{n h^d}\sum_{i=1}^n K\left(\frac{x-y_i}{h}\right)$, depending on the bandwidth $h$: a classical regime for large bandwidth where the Central Limit Theorem (CLT) holds, which is akin to the one found in traditional approaches. Below a certain value of the bandwidth, $h_{CLT}(α)$, we find that the CLT breaks down. The statistics of $\hatρ_h^{\mathcal {D}}(x)$ for a fixed $x$ drawn from $ρ(x)$ is given by a heavy-tailed distribution (an alpha-stable distribution). In particular below a value $h_G(α)$, we find that $\hatρ_h^{\mathcal {D}}(x)$ is governed by extreme value statistics: only a few points in the database matter and give the dominant contribution to the density estimator. We provide a detailed analysis for high-dimensional multivariate Gaussian data. We show that the optimal bandwidth threshold based on Kullback-Leibler divergence lies in the new statistical regime identified in this paper. As known by practitioners, when decreasing the bandwidth a Kernel-estimated estimated changes from a smooth curve to a collections of peaks centred on the data points. Our findings reveal that this general phenomenon is related to sharp transitions between phases characterized by different statistical properties, and offer new insights for Kernel density estimation in high-dimensional settings.

Figures (4)

• Figure 2.1: Numerical distribution of $\frac{1}{d}\log \hat{\rho}_h^{\mathcal{D}}(x)$ for for $n=164$, $d=51$ and $h=3$. The distribution is obtained generating $10^5$ samples of the datapoints $\{y_i\}$. The vertical line corresponds to the value of $\frac{1}{d}\log {\mathbb {E}}_{\mathcal{D}}[\hat{\rho}_h^{\mathcal{D}}(x)]$.
• Figure 2.2: Left: Numerical distribution of $\frac{1}{d}\log \hat{\rho}_h^{\mathcal{D}}(x)$ for for $n=164$, $d=51$ and $h=0.9$. The distribution is obtained generating $10^5$ samples of the datapoints $\{y_i\}$. The vertical line corresponds to the value of $\frac{1}{d}\log {\mathbb {E}}_{x_i}[\hat{\rho}_h^{\mathcal{D}}(x)]$. Right: Numerical distribution of $z=\hat{\rho}_h^{\mathcal{D}}(x)/\hat{\rho}_h^{typ}(x)$ in a log-log scale. The line corresponds to a power law with $z^{-1-m}$ with exponent $m=0.435715$ (obtained from our theory, see Sec. \ref{['sec:proba_distrib_Z']}).
• Figure 5.1: The case where $\rho$ is a Gaussian with identity covariance matrix. Left: $h_{CLT}^2$ (top curve) and $h_G^2$ are plotted as function of $\alpha$ for a kernel with $\gamma=1$. Right: For each of the three kernels with $\gamma=1,2,3$, plot of $D_{KL}$ versus $h$, comparison of the theory (red curves) and the experiment (blue curves) described in the text.
• Figure 6.1: The REM phase transition. We sketch the behaviour of $\alpha-I({\varepsilon})$ in the two phases. Left: the uncondensed phase; the slope at ${\varepsilon}={\varepsilon}_0$ is larger than one, the integral in \ref{['eq:REM_Zint']} is dominated by the point $\tilde{{\varepsilon}}$ which is inside the interval. Right :the condensed phase; the slope at ${\varepsilon}={\varepsilon}_0$ is larger than one; the integral in \ref{['eq:REM_Zint']} is dominated by the boundary ${\varepsilon}={\varepsilon}_0$

Theorems & Definitions (13)

• Definition 3.1: Pure densities
• Definition 3.2: Mixed densities
• Definition 3.3
• Proposition 4.1
• Lemma 4.2
• Theorem 4.3
• Corollary 4.4
• Corollary 4.5
• Corollary 4.6
• Proposition 5.1