Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast and Scalable Score-Based Kernel Calibration Tests

Pierre Glaser, David Widmann, Fredrik Lindsten, Arthur Gretton

TL;DR

This work tackles calibration testing for probabilistic models, including unnormalized densities, where traditional density-based expectations are expensive or intractable. It proposes the Kernel Calibration-Conditional Stein Discrepancy (KCCSD) test, a fast, score-based, kernel CGOF method that avoids density-expectation computations and provides statistically sound type-I error control. The authors introduce two kernel constructions grounded in a Generalized Fisher Divergence (GFD) and its kernelized diffusion variant (KGFD), along with a diffusion interpretation that links these kernels to stochastic processes and MMD, yielding universal, tractable estimators. The approach achieves calibrated performance on synthetic benchmarks and offers practical relevance for Bayesian inference and simulation-based problems, with potential to serve as a regularizer during model training. Overall, KCCSD delivers scalable calibration testing for complex probabilistic models, including unnormalized or energy-based densities, expanding reliable model assessment in SBI and related domains.

Abstract

We introduce the Kernel Calibration Conditional Stein Discrepancy test (KCCSD test), a non-parametric, kernel-based test for assessing the calibration of probabilistic models with well-defined scores. In contrast to previous methods, our test avoids the need for possibly expensive expectation approximations while providing control over its type-I error. We achieve these improvements by using a new family of kernels for score-based probabilities that can be estimated without probability density samples, and by using a conditional goodness-of-fit criterion for the KCCSD test's U-statistic. We demonstrate the properties of our test on various synthetic settings.

Fast and Scalable Score-Based Kernel Calibration Tests

TL;DR

This work tackles calibration testing for probabilistic models, including unnormalized densities, where traditional density-based expectations are expensive or intractable. It proposes the Kernel Calibration-Conditional Stein Discrepancy (KCCSD) test, a fast, score-based, kernel CGOF method that avoids density-expectation computations and provides statistically sound type-I error control. The authors introduce two kernel constructions grounded in a Generalized Fisher Divergence (GFD) and its kernelized diffusion variant (KGFD), along with a diffusion interpretation that links these kernels to stochastic processes and MMD, yielding universal, tractable estimators. The approach achieves calibrated performance on synthetic benchmarks and offers practical relevance for Bayesian inference and simulation-based problems, with potential to serve as a regularizer during model training. Overall, KCCSD delivers scalable calibration testing for complex probabilistic models, including unnormalized or energy-based densities, expanding reliable model assessment in SBI and related domains.

Abstract

We introduce the Kernel Calibration Conditional Stein Discrepancy test (KCCSD test), a non-parametric, kernel-based test for assessing the calibration of probabilistic models with well-defined scores. In contrast to previous methods, our test avoids the need for possibly expensive expectation approximations while providing control over its type-I error. We achieve these improvements by using a new family of kernels for score-based probabilities that can be estimated without probability density samples, and by using a conditional goodness-of-fit criterion for the KCCSD test's U-statistic. We demonstrate the properties of our test on various synthetic settings.
Paper Structure (37 sections, 7 theorems, 91 equations, 15 figures)

This paper contains 37 sections, 7 theorems, 91 equations, 15 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Notation
  4. Calibration of Predictive Models
  5. Kernel Conditional Goodness-of-Fit Test
  6. Kernel Calibration-Conditional Stein Discrepancy
  7. Tractable Kernels for General Unnormalized Densities
  8. General Recipe
  9. The Generalized Fisher Divergence (Kernel)
  10. The Exponentiated-GFD Kernel
  11. The Kernelized Generalized Fisher Divergence (kernel)
  12. A diffusion interpretation of the KGFD
  13. Fast and scalable calibration tests
  14. Experiments
  15. Mean Gaussian Model (MGM)
  16. ...and 22 more sections

Key Result

Proposition 3.1

Under weak assumptions (see Lemma B.1), the KCCSD with respect to kernels $l \colon \mathcal{Y} \times \mathcal{Y} \to \mathbb{R}$ and $k \colon P_{|\mathcal{X}} \times P_{|\mathcal{X}} \to \mathbb{R}$ is equivalent to the SKCE with kernel $H \colon (P_{|\mathcal{X}} \times \mathcal{Y}) \times (P_{|

Figures (15)

  • Figure 1: Rejection rates of the KCCSD and SKCE tests with a Gaussian kernel on the target space $\mathcal{Y}$ (significance level $\alpha = 0.05$). All kernels and test statistics are evaluated exactly using closed-form expressions.
  • Figure 2: False rejection rates of the SKCE tests for the calibrated LGM, HMC, and QGM ($n = 200$ data points, significance level $\alpha = 0.05$). The expectations in the test statistic are estimated with 2 samples obtained with the Metropolis-adjusted Langevin algorithm (MALA) without step size tuning.
  • Figure E.1: Relationships between the Fisher divergence, the KL divergence, the MMD, and the KSD liu2016short.
  • Figure F.1: False rejection rate of the KCCSD for MGM ($\delta = 0$).
  • Figure F.2: False rejection rate of the SKCE for MGM ($\delta = 0$).
  • ...and 10 more figures

Theorems & Definitions (14)

  • Proposition 3.1: Special case of Lemma B.1
  • Definition 4.1: Generalized Fisher Divergence
  • Definition 4.2: Exponentiated GFD Kernel
  • Proposition 4.1
  • proof
  • Definition 4.3: Exponentiated KGFD Kernel
  • Proposition 4.2
  • Proposition 4.3: Diffusion interpretation of the KGFD
  • proof
  • Definition 5.1: Conservativeness of a Bayesian model Hermans2021
  • ...and 4 more