Regularized $f$-Divergence Kernel Tests

TL;DR

The paper develops a kernel-based framework to construct two-sample tests from the entire family of $f$-divergences via a regularized variational representation, with adaptive kernel and regularization hyperparameters. It provides finite-sample convergence guarantees for the ratio and divergence estimators, and establishes permutation-based tests with controlled Type I error and non-asymptotic power bounds. By specializing to Hockey-Stick divergences, the authors demonstrate practical use in differential privacy auditing and machine unlearning, while also introducing regularized $f$-divergences (e.g., KALE, DrMMD) and an aggregation strategy ($f$-Agg) to robustly detect a range of alternatives. A key insight is that different $f$-divergences capture different localized departures from the null, motivating adaptive fusion and three-sample testing to address unlearning evaluation. Overall, the framework enables principled, adaptable, and powerful distributional testing with concrete applications to privacy and learning systems.

Abstract

We propose a framework to construct practical kernel-based two-sample tests from the family of $f$-divergences. The test statistic is computed from the witness function of a regularized variational representation of the divergence, which we estimate using kernel methods. The proposed test is adaptive over hyperparameters such as the kernel bandwidth and the regularization parameter. We provide theoretical guarantees for statistical test power across our family of $f$-divergence estimates. While our test covers a variety of $f$-divergences, we bring particular focus to the Hockey-Stick divergence, motivated by its applications to differential privacy auditing and machine unlearning evaluation. For two-sample testing, experiments demonstrate that different $f$-divergences are sensitive to different localized differences, illustrating the importance of leveraging diverse statistics. For machine unlearning, we propose a relative test that distinguishes true unlearning failures from safe distributional variations.

Figures (9)

• Figure 1: Performance on perturbed $d$-dimensional uniform alternatives with varying perturbation amplitudes. As the amplitude increases, the deviation from uniformity grows, leading to higher statistical power. Both MMD and DrMMD perform well in this setting, and the performance decreases for all methods in two dimensions. The $f$-Agg test maintains almost the same performance as the best method.
• Figure 2: Expo-1D null ($H_0$) and alternative hypothesis density functions varying the multiplier $k$.
• Figure 3: Detection rate comparison on the Expo-1D task, varying the multiplier $k$ of the perturbation. The DrMMD test achieves the highest power across all values.
• Figure 4: Performance on perturbed $d$-dimensional uniform alternatives with varying perturbation amplitudes for a diverse set of $f$-divergences. MMD outperforms all divergences in this setting. The performance decreases for all methods in two dimensions.
• Figure 5: DrMMD (Regularized $\chi^2$) and KL are the top-performing divergences overall in this setting. MMD improves in smooth settings (large $k$). For smaller $k$, the regularized estimators, KALE (regularized log-ratio) and DrMMD (regularized ratio) have superior performance.

Theorems & Definitions (29)

• Lemma 2.1: Witness convergence
• Lemma 2.2: $f$-Divergence convergence
• Theorem 3.1: Asymptotic Power
• Theorem 3.2: Non-asymptotic Power
• Definition 5.1
• Definition A.1
• Theorem A.2: Theorem 2, HG18
• Definition A.3
• Theorem D.1
• Lemma D.2