Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Kernel Tests of Equivalence

Xing Liu, Axel Gandy

Abstract

We propose novel kernel-based tests for assessing the equivalence between distributions. Traditional goodness-of-fit testing is inappropriate for concluding the absence of distributional differences, because failure to reject the null hypothesis may simply be a result of lack of test power, also known as the Type-II error. This motivates \emph{equivalence testing}, which aims to assess the \emph{absence} of a statistically meaningful effect under controlled error rates. However, existing equivalence tests are either limited to parametric distributions or focus only on specific moments rather than the full distribution. We address these limitations using two kernel-based statistical discrepancies: the \emph{kernel Stein discrepancy} and the \emph{Maximum Mean Discrepancy}. The null hypothesis of our proposed tests assumes the candidate distribution differs from the nominal distribution by at least a pre-defined margin, which is measured by these discrepancies. We propose two approaches for computing the critical values of the tests, one using an asymptotic normality approximation, and another based on bootstrapping. Numerical experiments are conducted to assess the performance of these tests.

Kernel Tests of Equivalence

Abstract

We propose novel kernel-based tests for assessing the equivalence between distributions. Traditional goodness-of-fit testing is inappropriate for concluding the absence of distributional differences, because failure to reject the null hypothesis may simply be a result of lack of test power, also known as the Type-II error. This motivates \emph{equivalence testing}, which aims to assess the \emph{absence} of a statistically meaningful effect under controlled error rates. However, existing equivalence tests are either limited to parametric distributions or focus only on specific moments rather than the full distribution. We address these limitations using two kernel-based statistical discrepancies: the \emph{kernel Stein discrepancy} and the \emph{Maximum Mean Discrepancy}. The null hypothesis of our proposed tests assumes the candidate distribution differs from the nominal distribution by at least a pre-defined margin, which is measured by these discrepancies. We propose two approaches for computing the critical values of the tests, one using an asymptotic normality approximation, and another based on bootstrapping. Numerical experiments are conducted to assess the performance of these tests.
Paper Structure (42 sections, 13 theorems, 83 equations, 6 figures, 1 table, 4 algorithms)

This paper contains 42 sections, 13 theorems, 83 equations, 6 figures, 1 table, 4 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Equivalence Testing
  4. Kernel-based Discrepancies
  5. Kernel Stein Discrepancy
  6. Maximum Mean Discrepancy
  7. Other Related Work
  8. One-Sample Equivalence Tests with KSD
  9. A Normal Test
  10. A Bootstrapped Test
  11. Two-Sample Equivalence Tests with MMD
  12. A Normal Test
  13. A Bootstrapped Test
  14. A Minimal-Effect Approach for Selecting the Equivalence Margin
  15. Experiments
  16. ...and 27 more sections

Key Result

Proposition 1

Suppose assump:kernel_universal, assump:score_condition and assump:ksd_kernel_condition hold.

Figures (6)

  • Figure 1: Comparison of different types of hypothesis testing. The shaded area depicts the space of distributions of interest. The null sets are colored in orange, and the alternative sets are colored in grey. Left. standard testing with a point null hypothesis. Middle. Robust testing based on a statistical discrepancy $D$. Right. Equivalence testing.
  • Figure 2: Gaussian mean-shift experiments with varying sample sizes. The black dotted vertical line is the equivalence margin $\theta$.
  • Figure 3: Gaussian mean-shift experiments with varying equivalence margins $\theta$, selected to be the population KSD values for different mean shifts.
  • Figure 4: Gaussian mean-shift experiments with $\theta$ selected using a power guarantee with $\beta = 0.2$. Top. KSD-based equivalence tests. Bottom. MMD-based equivalence tests.
  • Figure 5: Gaussian-Bernoulli RBM experiment with $\theta$ selected using a Type-II error guarantee with $\beta = 0.2$. The same results are plotted against the KSD values (left) and the standard deviations of the noise (right).
  • ...and 1 more figures

Theorems & Definitions (25)

  • Proposition 1: KSD asymptotics
  • Theorem 2: E-KSD-Normal test
  • Remark 2.1
  • Remark 2.2
  • Lemma 3
  • Remark 3.1
  • Remark 3.2
  • Theorem 4: E-KSD-Boot test
  • Remark 4.1
  • Proposition 5: MMD asymptotics
  • ...and 15 more