Kernel Embeddings and the Separation of Measure Phenomenon

TL;DR

The paper establishes a fundamental link between nonparametric two-sample testing and the mutual singularity of Gaussian embeddings in RKHSs. By leveraging kernel covariance embeddings, it shows that distinct non-atomic distributions become maximally separated when embedded as zero-mean Gaussian measures, with the covariance structure driving the separation. This separation-of-measure phenomenon reframes testing as a problem in the information geometry of Gaussian measures, illuminating why kernel methods can be highly effective in high-dimensional settings. The results hold under broad conditions (locally compact Polish spaces, non-atomic measures, universal kernels) and suggest refined embedding-based procedures that can outperform mean-embedding-only approaches in practice.

Abstract

We prove that kernel covariance embeddings lead to information-theoretically perfect separation of distinct continuous probability distributions. In statistical terms, we establish that testing for the \emph{equality} of two non-atomic (Borel) probability measures on a locally compact Polish space is \emph{equivalent} to testing for the \emph{singularity} between two centered Gaussian measures on a reproducing kernel Hilbert space. The corresponding Gaussians are defined via the notion of kernel covariance embedding of a probability measure, and the Hilbert space is that generated by the embedding kernel. Distinguishing singular Gaussians is structurally simpler from an information-theoretic perspective than non-parametric two-sample testing, particularly in complex or high-dimensional domains. This is because singular Gaussians are supported on essentially separate and affine subspaces. Our proof leverages the classical Feldman-Hájek dichotomy, and shows that even a small perturbation of a continuous distribution will be maximally magnified through its Gaussian embedding. This ``separation of measure phenomenon'' appears to be a blessing of infinite dimensionality, by means of embedding, with the potential to inform the design of efficient inference tools in considerable generality. The elicitation of this phenomenon also appears to crystallize, in a precise and simple mathematical statement, a core mechanism underpinning the empirical effectiveness of kernel methods.

Figures (3)

• Figure 1: Since Gaussians are always supported on affine sets, there is structure to the way singularity can manifest. In $\mathbb{R}^2$, for instance this can arise because the two Gaussians are supported on distinct lines (a) or because one is supported on the (full) plane, while the other on a line. In $\mathbb{R}^3$, mutual singularity of Gaussians can arise, for instance, when the measures are supported on distinct planes.
• Figure 2: Gaussian embeddings magnify distributional differences in a structured fashion: distinct measures on $\cX$ ($\bbP,\bbQ$ on the left) are mapped to mutually singular Gaussian measures on $\cH$ ($\cN_\bbP,\cN_\bbQ$ on the right, where $\cN_\bbP, \cN_\bbQ$ are either centered or uncentered Gaussian embeddings of $\bbP,\bbQ$).
• Figure 3: Monte Carlo Illustration of the sampling behaviour of MMD and $\mathsf{D}_{\mathsf{KL}}$ under null ($\alpha=1/2,\varepsilon=0$) and alternative ($\alpha=1/2,\varepsilon=1/4$) regimes, using a Laplacian kernel calibrated by median heuristic. Left: Smoothed Monte Carlo sampling distributions under $H_0$ and $H_1$, centred to for ease of comparison. Right: Proportion of realisations under $H_1$ where each statistic exceeds the 95th percentile of the null sampling distribution based on $K = 100$ runs.

Theorems & Definitions (14)

• theorem 1
• corollary 1
• theorem 2
• remark 1
• proposition 1
• lemma 1
• proof : Proof of Lemma \ref{['lem:Semb_is_multop']}
• lemma 2
• proof : Proof of Lemma \ref{['lem:multisnotcompact']}
• proof : Proof of Theorem \ref{['thm:main']}