Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Composite Goodness-of-fit Tests with Kernels

Oscar Key, Arthur Gretton, François-Xavier Briol, Tamara Fernandez

TL;DR

The paper tackles the problem of model misspecification in probabilistic modeling by introducing kernel-based composite goodness-of-fit tests that decide whether data come from any distribution within a parametric family {P_θ}. The main approach combines minimum distance estimation with kernel discrepancies (primarily MMD^2, with discussion of KSD) to jointly estimate the closest model and test goodness-of-fit on the same data, using bootstrap methods to set thresholds without data-splitting. Theoretical results show the MMD-based statistic has a well-behaved null distribution and consistent power under the alternative, even when estimation and testing use the same data, and practical implementations via wild and parametric bootstraps are explored. Empirical studies on Gaussian models, toggle-switch networks, and kernel-exponential-family density estimation demonstrate the method’s applicability to unnormalised and simulator-based models, while highlighting limitations related to kernel choice, optimization stability, and computational cost. The work bridges kernel-based testing with classical likelihood-ratio and characteristic-function approaches, providing a versatile toolkit for assessing model adequacy in complex, high-dimensional settings. The proposed framework offers a principled way to quantify misspecification and guide model refinement in practical scientific and engineering applications.

Abstract

Model misspecification can create significant challenges for the implementation of probabilistic models, and this has led to development of a range of robust methods which directly account for this issue. However, whether these more involved methods are required will depend on whether the model is really misspecified, and there is a lack of generally applicable methods to answer this question. In this paper, we propose one such method. More precisely, we propose kernel-based hypothesis tests for the challenging composite testing problem, where we are interested in whether the data comes from any distribution in some parametric family. Our tests make use of minimum distance estimators based on the maximum mean discrepancy and the kernel Stein discrepancy. They are widely applicable, including whenever the density of the parametric model is known up to normalisation constant, or if the model takes the form of a simulator. As our main result, we show that we are able to estimate the parameter and conduct our test on the same data (without data splitting), while maintaining a correct test level. Our approach is illustrated on a range of problems, including testing for goodness-of-fit of an unnormalised non-parametric density model, and an intractable generative model of a biological cellular network.

Composite Goodness-of-fit Tests with Kernels

TL;DR

The paper tackles the problem of model misspecification in probabilistic modeling by introducing kernel-based composite goodness-of-fit tests that decide whether data come from any distribution within a parametric family {P_θ}. The main approach combines minimum distance estimation with kernel discrepancies (primarily MMD^2, with discussion of KSD) to jointly estimate the closest model and test goodness-of-fit on the same data, using bootstrap methods to set thresholds without data-splitting. Theoretical results show the MMD-based statistic has a well-behaved null distribution and consistent power under the alternative, even when estimation and testing use the same data, and practical implementations via wild and parametric bootstraps are explored. Empirical studies on Gaussian models, toggle-switch networks, and kernel-exponential-family density estimation demonstrate the method’s applicability to unnormalised and simulator-based models, while highlighting limitations related to kernel choice, optimization stability, and computational cost. The work bridges kernel-based testing with classical likelihood-ratio and characteristic-function approaches, providing a versatile toolkit for assessing model adequacy in complex, high-dimensional settings. The proposed framework offers a principled way to quantify misspecification and guide model refinement in practical scientific and engineering applications.

Abstract

Model misspecification can create significant challenges for the implementation of probabilistic models, and this has led to development of a range of robust methods which directly account for this issue. However, whether these more involved methods are required will depend on whether the model is really misspecified, and there is a lack of generally applicable methods to answer this question. In this paper, we propose one such method. More precisely, we propose kernel-based hypothesis tests for the challenging composite testing problem, where we are interested in whether the data comes from any distribution in some parametric family. Our tests make use of minimum distance estimators based on the maximum mean discrepancy and the kernel Stein discrepancy. They are widely applicable, including whenever the density of the parametric model is known up to normalisation constant, or if the model takes the form of a simulator. As our main result, we show that we are able to estimate the parameter and conduct our test on the same data (without data splitting), while maintaining a correct test level. Our approach is illustrated on a range of problems, including testing for goodness-of-fit of an unnormalised non-parametric density model, and an intractable generative model of a biological cellular network.
Paper Structure (46 sections, 22 theorems, 148 equations, 15 figures, 3 algorithms)

This paper contains 46 sections, 22 theorems, 148 equations, 15 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Kernel-based Discrepancies
  4. Goodness-of-fit Testing with Kernels
  5. Minimum Distance Estimation with Kernels
  6. Methodology: The Test Statistic
  7. Composite Goodness-of-fit Testing with Kernels
  8. Theoretical Analysis of the MMD Composite Goodness-of-fit fest
  9. Assumptions
  10. Behaviour of the Test Statistic Under $H^{C}_0$ and $H^{C}_1$
  11. Methodology: Practical Implementation via Bootstraps
  12. The Wild Bootstrap
  13. The Parametric Bootstrap
  14. Empirical Results
  15. Gaussian Model
  16. ...and 31 more sections

Key Result

Theorem 2

Under $H^{C}_0$ and condition:observationscondition:domainscondition:modelcondition:parameter_nullcondition:mmd_kernelcondition:invertible: where $(Z_i)_{i\geq 1}$ are a collection of i.i.d. standard normal random variables and $(\lambda_i)_{i \geq 1}$ are constants, which depend on the choice of kernel, where $\sum_{i=1}^\infty \lambda_i < \infty$.

Figures (15)

  • Figure 1: Illustration of the sources of error when estimating the test statistic under the null hypothesis. Existing non-composite tests use the test statistic $n\mathop{\mathrm{D}}\nolimits(P,Q_n)$, where $\mathop{\mathrm{D}}\nolimits$ is a statistical divergence, and thus encounter only "test" error. The composite test we introduce uses the statistic $n\mathop{\mathrm{D}}\nolimits(P_{\hat{\theta}_n},Q_n)$, and thus encounters both "estimation" and "test" error.
  • Figure 2: Distribution of $\mathop{\mathrm{MMD}}\nolimits^2(P_{\hat{\theta}_n}, Q_n)$ and $\mathop{\mathrm{MMD}}\nolimits^2_W(P_{\hat{\theta}_n,n}, Q_n)$ under $H^{C}_0$, obtained by simulation. and show the respective $(1-\alpha)$ quantiles, demonstrating that the wild bootstrap estimates a conservative threshold.
  • Figure 3: Performance of the MMD and KSD tests as $n$ increases, where $H^{C}_0 : P = \mathop{\mathrm{\mathcal{N}}}\nolimits(\mu, 1)$. show the parametric bootstrap, show the wild bootstrap, shows the level. The error bars show one standard error across $4$ random seeds.
  • Figure 4: Power of the parametric bootstrap tests as $d$ increases. $H^{C}_0 : P = \mathop{\mathrm{\mathcal{N}}}\nolimits(\mu, \sigma^2)$, $H^C_{1a}$ and $H^C_{1b}$ indicate $Q = \operatorname{Students-t}$, with $3$ and $5$ degrees of freedom respectively. shows the level.
  • Figure 5: Comparison of the MMD tests against Wolfer2022. $H^{C}_0 : P = \mathop{\mathrm{\mathcal{N}}}\nolimits(\mu, 1)$ and $Q = \mathop{\mathrm{\mathcal{N}}}\nolimits(\mu_0, \sigma^2)$. $n=200$. shows the level. The error bars show one standard error across $4$ random seeds.
  • ...and 10 more figures

Theorems & Definitions (25)

  • Remark 1
  • Theorem 2: Convergence under the null hypothesis
  • Theorem 3: Consistency under the alternative hypothesis
  • Theorem 4
  • Theorem 5: Parametric bootstrap under the null
  • Theorem 6: Theorem 3 and Corollary 4, Sriperumbudur2009
  • Theorem 7
  • Remark 8: Bound for RKHS functions
  • Remark 9: Interchange of integration and differentiation
  • Theorem 10
  • ...and 15 more