Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Score Change of Variables

Stephen Robbins

TL;DR

This work derives a general score change of variables formula that expresses the transformed score $\nabla_{\mathbf{y}} \log q(\mathbf{y})$ directly from the original score $\nabla_{\mathbf{x}} \log p(\mathbf{x})$ and the derivatives of a smooth invertible map $\phi$. It then harnesses this result to (i) establish a reverse-time Itô lemma for transformed diffusion processes, enabling sampling in a transformed space using scores learned in the original space, and (ii) introduce Generalized Sliced Score Matching (GSSM), which extends sliced score matching to arbitrary smooth transformations. The paper demonstrates two key applications: diffusion on the probability simplex via an additive logistic transform, and density estimation with Deep Kernel Exponential Families using GSSM, including a variance-reduced variant. Together these contributions provide a principled framework for transforming score-based methods across geometries, enabling training in one space while sampling in another, and broadening the flexibility and applicability of score-based density estimation and generative modeling.

Abstract

We derive a general change of variables formula for score functions, showing that for a smooth, invertible transformation $\mathbf{y} = φ(\mathbf{x})$, the transformed score function $\nabla_{\mathbf{y}} \log q(\mathbf{y})$ can be expressed directly in terms of $\nabla_{\mathbf{x}} \log p(\mathbf{x})$. Using this result, we develop two applications: First, we establish a reverse-time Itô lemma for score-based diffusion models, allowing the use of $\nabla_{\mathbf{x}} \log p_t(\mathbf{x})$ to reverse an SDE in the transformed space without directly learning $\nabla_{\mathbf{y}} \log q_t(\mathbf{y})$. This approach enables training diffusion models in one space but sampling in another, effectively decoupling the forward and reverse processes. Second, we introduce generalized sliced score matching, extending traditional sliced score matching from linear projections to arbitrary smooth transformations. This provides greater flexibility in high-dimensional density estimation. We demonstrate these theoretical advances through applications to diffusion on the probability simplex and empirically compare our generalized score matching approach against traditional sliced score matching methods.

Score Change of Variables

TL;DR

This work derives a general score change of variables formula that expresses the transformed score directly from the original score and the derivatives of a smooth invertible map . It then harnesses this result to (i) establish a reverse-time Itô lemma for transformed diffusion processes, enabling sampling in a transformed space using scores learned in the original space, and (ii) introduce Generalized Sliced Score Matching (GSSM), which extends sliced score matching to arbitrary smooth transformations. The paper demonstrates two key applications: diffusion on the probability simplex via an additive logistic transform, and density estimation with Deep Kernel Exponential Families using GSSM, including a variance-reduced variant. Together these contributions provide a principled framework for transforming score-based methods across geometries, enabling training in one space while sampling in another, and broadening the flexibility and applicability of score-based density estimation and generative modeling.

Abstract

We derive a general change of variables formula for score functions, showing that for a smooth, invertible transformation , the transformed score function can be expressed directly in terms of . Using this result, we develop two applications: First, we establish a reverse-time Itô lemma for score-based diffusion models, allowing the use of to reverse an SDE in the transformed space without directly learning . This approach enables training diffusion models in one space but sampling in another, effectively decoupling the forward and reverse processes. Second, we introduce generalized sliced score matching, extending traditional sliced score matching from linear projections to arbitrary smooth transformations. This provides greater flexibility in high-dimensional density estimation. We demonstrate these theoretical advances through applications to diffusion on the probability simplex and empirically compare our generalized score matching approach against traditional sliced score matching methods.

Paper Structure

This paper contains 69 sections, 5 theorems, 145 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Main Result
  3. Assumptions
  4. Score Change of Variables
  5. Special Cases
  6. Applications
  7. Reverse-Time Itô Lemma
  8. Transformed Score Matching
  9. Background: Score Matching
  10. Sliced Score Matching
  11. Generalized Sliced Score Matching
  12. Derivation Outline
  13. Examples
  14. Diffusion on the Probability Simplex for Chess Positions
  15. Representing Chess Positions
  16. ...and 54 more sections

Key Result

Theorem 1

Suppose Assumption assumption:smooth_bijectivity_support_multidim holds. Then, the score function $\nabla_{\mathbf{y}} \log q(\mathbf{y})$ can be expressed as:

Figures (6)

  • Figure 1: $w = 0.8$.
  • Figure 2: $w = 0.9$.
  • Figure 3: $w = 1.0$.
  • Figure 4: $w = 1.1$.
  • Figure 6: Exact Score Matching loss across three UCI datasets; lower loss is better.
  • ...and 1 more figures

Theorems & Definitions (13)

  • Theorem 1: Score Change of Variables: $\mathbb{R}^n \to \mathbb{R}^n$
  • Remark 1
  • proof
  • Corollary 1: Score Change of Variables: $\mathbb{R} \to \mathbb{R}$
  • Corollary 2: Score Change of Variables: $\mathbb{R}^n \to \mathbb{R}$
  • proof
  • Corollary 3: Reverse-Time Itô Lemma
  • proof
  • Remark 2
  • proof
  • ...and 3 more