Approximating $f$-Divergences with Rank Statistics

TL;DR

The paper tackles the challenge of estimating $f$-divergences from samples without density-ratio estimation by introducing a rank-statistic framework based on the probability integral transform and rank histograms. It defines a univariate rank-divergence $D^{(K)}_{f,\nu}(\mu)$ that is convex, monotone in the resolution $K$, and provides convergence guarantees as $K\to\infty$, along with finite-sample deviation bounds and asymptotic normality. The authors extend the approach to high dimensions via sliced divergences, proving similar properties for the averaged one-dimensional projections and establishing consistency for the sliced limit. They validate the method empirically against neural baselines and demonstrate its utility as a learning objective for generative modelling, including experiments on CelebA, while outlining practical limitations and future research directions.

Abstract

We introduce a rank-statistic approximation of $f$-divergences that avoids explicit density-ratio estimation by working directly with the distribution of ranks. For a resolution parameter $K$, we map the mismatch between two univariate distributions $μ$ and $ν$ to a rank histogram on $\{ 0, \ldots, K\}$ and measure its deviation from uniformity via a discrete $f$-divergence, yielding a rank-statistic divergence estimator. We prove that the resulting estimator of the divergence is monotone in $K$, is always a lower bound of the true $f$-divergence, and we establish quantitative convergence rates for $K\to\infty$ under mild regularity of the quantile-domain density ratio. To handle high-dimensional data, we define the sliced rank-statistic $f$-divergence by averaging the univariate construction over random projections, and we provide convergence results for the sliced limit as well. We also derive finite-sample deviation bounds along with asymptotic normality results for the estimator. Finally, we empirically validate the approach by benchmarking against neural baselines and illustrating its use as a learning objective in generative modelling experiments.

Figures (10)

• Figure 1: Conceptual illustration of the rank-statistic $f$-divergence. (a) When $\mu = \nu$, samples are uniformly interleaved, resulting in a uniform rank histogram. (b) With a mismatch, samples from $\mu$ cluster in specific rank bins, creating a non-uniform histogram that indicates divergence.
• Figure 2: Convergence of Kullback--Leibler divergence estimates for increasing sample size $n$, averaged over 10 independent runs. Shaded bands denote the $\pm 1$ standard deviation interval. Results are shown for $K = 64$ across all samples.
• Figure 3: Comparison of mean shift metrics across dimensions for KL, Hellinger, and JS divergences.
• Figure 4: Rank-Proximal Transport on 2D toy targets. Using SGD to minimize the rank-statistic KL with $L=10$ random projections, particles (orange) evolve from a Gaussian start ($t=0$) to match the target support (blue).
• Figure 5: CO-RPT samples on CelebA ($64\times64$) after $T=20{,}000$ outer steps.

Theorems & Definitions (43)

• Definition 2.1
• Definition 2.2
• Example 2.1
• Theorem 2.3
• proof
• Remark 2.4: Markov kernel interpretation of $D_{f}^{(K)}$
• Theorem 2.5: Convergence of the truncated divergence
• proof
• Example 2.2: Applicability of Convergence Rates
• Theorem 2.6: Univariate finite sample complexity