Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Implicit score matching meets denoising score matching: improved rates of convergence and log-density Hessian estimation

Konstantin Yakovlev, Anna Markovich, Nikita Puchkin

TL;DR

The paper addresses estimating the log-density gradient $s^*(y)$ using implicit and denoising score matching under data with low intrinsic dimension. It leverages weighted Gagliardo–Nirenberg inequalities to relate function norms and derivatives, enabling sharp convergence guarantees for both score estimators and their Jacobians. Remarkably, implicit score matching attains the same $\mathcal{O}(n^{-2\beta/(2\beta+d)})$ rates as denoising score matching, with the effective dimension $d$ rather than the ambient dimension driving the rate, and it shows that the log-density Hessian can be estimated by differentiating the score estimates, mitigating the curse of dimensionality. These results underpin the theoretical justification for the convergence of ODE-based samplers in diffusion models and provide a framework for statistically efficient estimation of higher-order derivatives in high dimensions.

Abstract

We study the problem of estimating the score function using both implicit score matching and denoising score matching. Assuming that the data distribution exhibiting a low-dimensional structure, we prove that implicit score matching is able not only to adapt to the intrinsic dimension, but also to achieve the same rates of convergence as denoising score matching in terms of the sample size. Furthermore, we demonstrate that both methods allow us to estimate log-density Hessians without the curse of dimensionality by simple differentiation. This justifies convergence of ODE-based samplers for generative diffusion models. Our approach is based on Gagliardo-Nirenberg-type inequalities relating weighted $L^2$-norms of smooth functions and their derivatives.

Implicit score matching meets denoising score matching: improved rates of convergence and log-density Hessian estimation

TL;DR

The paper addresses estimating the log-density gradient using implicit and denoising score matching under data with low intrinsic dimension. It leverages weighted Gagliardo–Nirenberg inequalities to relate function norms and derivatives, enabling sharp convergence guarantees for both score estimators and their Jacobians. Remarkably, implicit score matching attains the same rates as denoising score matching, with the effective dimension rather than the ambient dimension driving the rate, and it shows that the log-density Hessian can be estimated by differentiating the score estimates, mitigating the curse of dimensionality. These results underpin the theoretical justification for the convergence of ODE-based samplers in diffusion models and provide a framework for statistically efficient estimation of higher-order derivatives in high dimensions.

Abstract

We study the problem of estimating the score function using both implicit score matching and denoising score matching. Assuming that the data distribution exhibiting a low-dimensional structure, we prove that implicit score matching is able not only to adapt to the intrinsic dimension, but also to achieve the same rates of convergence as denoising score matching in terms of the sample size. Furthermore, we demonstrate that both methods allow us to estimate log-density Hessians without the curse of dimensionality by simple differentiation. This justifies convergence of ODE-based samplers for generative diffusion models. Our approach is based on Gagliardo-Nirenberg-type inequalities relating weighted -norms of smooth functions and their derivatives.
Paper Structure (24 sections, 18 theorems, 307 equations)

This paper contains 24 sections, 18 theorems, 307 equations.

Table of Contents

  1. Introduction
  2. Preliminaries and notations
  3. Results
  4. Weighted Gagliardo-Nirenberg inequalities for $\mathsf{p}^*$
  5. Score estimation using implicit score matching
  6. Score estimation using denoising score matching
  7. Proof of Theorem \ref{['th:estimation']}
  8. Proof of Lemma \ref{['lem:bernstein_cond_div']}
  9. Proof of Lemma \ref{['lem:bernstein_cond_helper']}
  10. Proofs of supplemental results from Section \ref{['sec:estimation_proof']}
  11. Proof of Lemma \ref{['lem:bernstein_cond']}
  12. Proof of Lemma \ref{['lem:psi1_norm_bound']}
  13. Proof of Lemma \ref{['lem:loss_cov_num_bound_true']}
  14. Proof of Lemma \ref{['lem:exp_max_log_data']}
  15. Proof of Lemma \ref{['lem:gelu_func_div_prox']}
  16. ...and 9 more sections

Key Result

Lemma 3.2

Suppose Assumption asn:relax_man holds. Let also $h : \mathbb R^D \to \mathbb R^D$ satisfy $|h|_{W^{\alpha, 2}(\mathsf{p}^*)} < \infty$ for some integer $\alpha \geqslant 2$. Then it holds that

Theorems & Definitions (24)

  • Definition 2.1: Hölder class
  • Definition 2.2: Sobolev space
  • Lemma 3.2
  • Definition 3.3: class of score estimators
  • Theorem 3.5: implicit score matching generalization bound
  • Definition 3.6
  • Theorem 3.7: denoising score matching generalization bound
  • Lemma 4.1
  • Lemma 4.2
  • Lemma 4.3
  • ...and 14 more