A Kernel Method for the Two-Sample Problem

TL;DR

The paper tackles the two-sample problem by introducing the Maximum Mean Discrepancy (MMD), a kernel-based statistic that measures distribution differences via RKHS mean embeddings. It develops three nonparametric tests: two with finite-sample guarantees from uniform convergence bounds and a third based on the asymptotic distribution, along with a linear-time variant for large-scale data. The framework unifies classical distribution metrics and extends to high-dimensional and structured domains such as graphs, enabling robust, scalable hypothesis testing without density estimation. Empirical results in data integration and related domains demonstrate strong performance and practical applicability of MMD-based tests across diverse data types.

Abstract

We propose a framework for analyzing and comparing distributions, allowing us to design statistical tests to determine if two samples are drawn from different distributions. Our test statistic is the largest difference in expectations over functions in the unit ball of a reproducing kernel Hilbert space (RKHS). We present two tests based on large deviation bounds for the test statistic, while a third is based on the asymptotic distribution of this statistic. The test statistic can be computed in quadratic time, although efficient linear time approximations are available. Several classical metrics on distributions are recovered when the function space used to compute the difference in expectations is allowed to be more general (eg. a Banach space). We apply our two-sample tests to a variety of problems, including attribute matching for databases using the Hungarian marriage method, where they perform strongly. Excellent performance is also obtained when comparing distributions over graphs, for which these are the first such tests.

Figures (4)

• Figure 1: Illustration of the function maximizing the mean discrepancy in the case where a Gaussian is being compared with a Laplace distribution. Both distributions have zero mean and unit variance. The function $f$ that witnesses the MMD has been scaled for plotting purposes, and was computed empirically on the basis of $2\times 10^4$ samples, using a Gaussian kernel with $\sigma=0.5$.
• Figure 2: Left: Empirical distribution of the MMD under $\mathcal{H}_0$, with $p$ and $q$ both Gaussians with unit standard deviation, using 50 samples from each. Right: Empirical distribution of the MMD under $\mathcal{H}_1$, with $p$ a Laplace distribution with unit standard deviation, and $q$ a Laplace distribution with standard deviation $3 \sqrt{2}$, using 100 samples from each. In both cases, the histograms were obtained by computing 2000 independent instances of the MMD.
• Figure 3: Illustration of the function maximizing the mean discrepancy when MMD is used as a measure of independence. A sample from dependent random variables $x$ and $y$ is shown in black, and the associated function $f$ that witnesses the MMD is plotted as a contour. The latter was computed empirically on the basis of $200$ samples, using a Gaussian kernel with $\sigma=0.2$.
• Figure 4: Type II performance of the various tests when separating two Gaussians, with test level $\alpha=0.05$. A Gaussians have same variance and different means. B Gaussians have same mean and different variances.

Theorems & Definitions (27)

• Lemma 1
• Definition 2
• Theorem 3
• Lemma 4
• Lemma 5
• Definition 6
• Theorem 7
• Theorem 8
• Remark 9: Reduction to Binary Classification
• Proposition 10