Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalized Score Matching: Bridging $f$-Divergence and Statistical Estimation Under Correlated Noise

Yirong Shen, Lu Gan, Cong Ling

TL;DR

This work generalizes score matching by introducing $f$-score matching for correlated, non-isotropic noise through a vector Gaussian channel. It establishes a fundamental link: the gradient of the $f$-divergence between output densities with respect to the noise covariance $oldsymbol{\Sigma}$ equals minus one-half the $f$-Fisher information, extending the classical score-matching and De Bruijn identities to broader noise structures. The special case of KL divergence recovers the standard relation to Fisher information and offers a high-dimensional de Bruijn identity for mismatched vector Gaussian channels, with implications for diffusion-based generative modeling and robust density estimation. Overall, the framework unifies and extends existing estimation objectives under correlated noise, enabling principled optimization (e.g., gradient or Newton steps) and providing theoretical foundations for diffusion-type models under realistic noise. The results pave the way for more robust generative modeling, covariance-structured learning, and informed training objectives beyond isotropic-noise assumptions.$

Abstract

Relative Fisher information, also known as score matching, is a recently introduced learning method for parameter estimation. Fundamental relations between relative entropy and score matching have been established in the literature for scalar and isotropic Gaussian channels. This paper demonstrates that such relations hold for a much larger class of observation models. We introduce the vector channel where the perturbation is non-isotropic Gaussian noise. For such channels, we derive new representations that connect the $f$-divergence between two distributions to the estimation loss induced by mismatch at the decoder. This approach not only unifies but also greatly extends existing results from both the isotropic Gaussian and classical relative entropy frameworks. Building on this generalization, we extend De Bruijn's identity to mismatched non-isotropic Gaussian models and demonstrate that the connections to generative models naturally follow as a consequence application of this new result.

Generalized Score Matching: Bridging $f$-Divergence and Statistical Estimation Under Correlated Noise

TL;DR

This work generalizes score matching by introducing -score matching for correlated, non-isotropic noise through a vector Gaussian channel. It establishes a fundamental link: the gradient of the -divergence between output densities with respect to the noise covariance equals minus one-half the -Fisher information, extending the classical score-matching and De Bruijn identities to broader noise structures. The special case of KL divergence recovers the standard relation to Fisher information and offers a high-dimensional de Bruijn identity for mismatched vector Gaussian channels, with implications for diffusion-based generative modeling and robust density estimation. Overall, the framework unifies and extends existing estimation objectives under correlated noise, enabling principled optimization (e.g., gradient or Newton steps) and providing theoretical foundations for diffusion-type models under realistic noise. The results pave the way for more robust generative modeling, covariance-structured learning, and informed training objectives beyond isotropic-noise assumptions.$

Abstract

Relative Fisher information, also known as score matching, is a recently introduced learning method for parameter estimation. Fundamental relations between relative entropy and score matching have been established in the literature for scalar and isotropic Gaussian channels. This paper demonstrates that such relations hold for a much larger class of observation models. We introduce the vector channel where the perturbation is non-isotropic Gaussian noise. For such channels, we derive new representations that connect the -divergence between two distributions to the estimation loss induced by mismatch at the decoder. This approach not only unifies but also greatly extends existing results from both the isotropic Gaussian and classical relative entropy frameworks. Building on this generalization, we extend De Bruijn's identity to mismatched non-isotropic Gaussian models and demonstrate that the connections to generative models naturally follow as a consequence application of this new result.
Paper Structure (14 sections, 4 theorems, 39 equations)

This paper contains 14 sections, 4 theorems, 39 equations.

Table of Contents

  1. Introduction
  2. Background
  3. Relative Fisher Information Matrix
  4. Learning Metric
  5. Probabilistic Diffusion Models
  6. Main Results
  7. Generalized Score Matching
  8. Classical Relative Entropy
  9. Concluding Remarks
  10. Appendix
  11. Derivative w.r.t. $\mathbf{\Sigma}$
  12. First derivative w.r.t. $\mathbf{y}$
  13. Hessian (second derivative) w.r.t. $\mathbf{y}$
  14. Relation between $\nabla_{\mathbf{\Sigma}} p(\mathbf{y})$ and $\mathbb{H}_\mathbf{y} p(\mathbf{y})$

Key Result

Theorem 1

Consider the signal model in vector channel, where $\mathbf{X}$ (with finite second-order moments) is arbitrary, and $\mathbf{N}$ is Gaussian with covariance matrix $\mathbf{\Sigma}$, independent of $\mathbf{X}$. Denote by $p(\mathbf{y})$ and $q(\mathbf{y};{\boldsymbol{\theta}})$ the densities of $\

Theorems & Definitions (10)

  • Definition 1
  • Definition 2
  • Theorem 1
  • Lemma 1
  • Lemma 2
  • proof
  • Corollary 1
  • proof
  • proof : Proof of Lemma \ref{['heat equation']}
  • proof : Proof of Lemma \ref{['integration vanish']}