Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space

Fangyikang Wang, Hubery Yin, Shaobin Zhuang, Huminhao Zhu, Yinan Li, Lei Qian, Chao Zhang, Hanbin Zhao, Hui Qian, Chen Li

TL;DR

This work reframes diffusion Fisher information as a weighted outer-product sum, proving that $\mathbf{F}_t(\mathbf{x}_t,t)$ resides in an outer-product span determined by the initial data and the noise schedule. It then introduces two efficient access methods: diffusion Fisher trace matching (DF-TM) for trace computation and diffusion Fisher endpoint approximation (DF-EA) for gradient-free adjoint operations, with rigorous error bounds. The methods yield accurate, faster likelihood evaluation and adjoint optimization compared to gradient-based VJP, and enable a numerical verification of the optimal transport property of PF-ODE maps under various initial data types. Collectively, the paper advances practical DF usage in diffusion models and deepens theoretical understanding of their transport properties, with code available at the provided repository.

Abstract

Recent Diffusion models (DMs) advancements have explored incorporating the second-order diffusion Fisher information (DF), defined as the negative Hessian of log density, into various downstream tasks and theoretical analysis. However, current practices typically approximate the diffusion Fisher by applying auto-differentiation to the learned score network. This black-box method, though straightforward, lacks any accuracy guarantee and is time-consuming. In this paper, we show that the diffusion Fisher actually resides within a space spanned by the outer products of score and initial data. Based on the outer-product structure, we develop two efficient approximation algorithms to access the trace and matrix-vector multiplication of DF, respectively. These algorithms bypass the auto-differentiation operations with time-efficient vector-product calculations. Furthermore, we establish the approximation error bounds for the proposed algorithms. Experiments in likelihood evaluation and adjoint optimization demonstrate the superior accuracy and reduced computational cost of our proposed algorithms. Additionally, based on the novel outer-product formulation of DF, we design the first numerical verification experiment for the optimal transport property of the general PF-ODE deduced map.

Efficiently Access Diffusion Fisher: Within the Outer Product Span Space

TL;DR

This work reframes diffusion Fisher information as a weighted outer-product sum, proving that resides in an outer-product span determined by the initial data and the noise schedule. It then introduces two efficient access methods: diffusion Fisher trace matching (DF-TM) for trace computation and diffusion Fisher endpoint approximation (DF-EA) for gradient-free adjoint operations, with rigorous error bounds. The methods yield accurate, faster likelihood evaluation and adjoint optimization compared to gradient-based VJP, and enable a numerical verification of the optimal transport property of PF-ODE maps under various initial data types. Collectively, the paper advances practical DF usage in diffusion models and deepens theoretical understanding of their transport properties, with code available at the provided repository.

Abstract

Recent Diffusion models (DMs) advancements have explored incorporating the second-order diffusion Fisher information (DF), defined as the negative Hessian of log density, into various downstream tasks and theoretical analysis. However, current practices typically approximate the diffusion Fisher by applying auto-differentiation to the learned score network. This black-box method, though straightforward, lacks any accuracy guarantee and is time-consuming. In this paper, we show that the diffusion Fisher actually resides within a space spanned by the outer products of score and initial data. Based on the outer-product structure, we develop two efficient approximation algorithms to access the trace and matrix-vector multiplication of DF, respectively. These algorithms bypass the auto-differentiation operations with time-efficient vector-product calculations. Furthermore, we establish the approximation error bounds for the proposed algorithms. Experiments in likelihood evaluation and adjoint optimization demonstrate the superior accuracy and reduced computational cost of our proposed algorithms. Additionally, based on the novel outer-product formulation of DF, we design the first numerical verification experiment for the optimal transport property of the general PF-ODE deduced map.

Paper Structure

This paper contains 51 sections, 18 theorems, 56 equations, 9 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Diffusion Models and Diffusion SDEs
  4. Diffusion Models Inference as Neural ODE
  5. Fisher Information in Diffusion Models
  6. Diffusion Fisher in Outer-Products Span
  7. The General Setting
  8. Diffusion Fisher Trace Matching
  9. Log-Likelihood in DMs
  10. Log-Likelihood Evaluation via VJP
  11. Log-Likelihood Evaluation via DF trace matching
  12. Experiments
  13. Diffusion Fisher Endpoint Approximation
  14. Adjoint optimization sampling.
  15. Adjoint ODE via VJP.
  16. ...and 36 more sections

Key Result

Proposition 1

Defines $v_i({\bm{x}}_t, t)$ as $\exp(-\frac{|{\bm{x}}_t - \alpha_t {\bm{y}}_i|^2}{2\sigma_t^2})\in {\mathbb{R}}$ and $w_i({\bm{x}}_t, t)$ as $\frac{v_i({\bm{x}}_t, t)}{\sum_j v_j({\bm{x}}_t, t)} \in {\mathbb{R}}$. If $q_0$ takes the form as in equation equation dirac_initial_dist, the diffusion Fis where we have simplified $w_i({\bm{x}}_t, t)$ to $w_i$, as it does not lead to any confusion.

Figures (9)

  • Figure 1:
  • Figure 2:
  • Figure 4: Our DF-TM method facilitates the effective evaluation of the NLL of generated samples with varying seeds. It can be demonstrated that a lower NLL signifies a region of higher possibility, thereby consistently indicating superior image quality.
  • Figure 5: SAC Aesthetic Score ($\uparrow$)
  • Figure 6: AVA Aesthetic Score ($\uparrow$)
  • ...and 4 more figures

Theorems & Definitions (25)

  • Proposition 1
  • Proposition 2
  • Proposition 3
  • Proposition 4
  • Proposition 5
  • Proposition 6
  • Proposition 7
  • Proposition 8
  • Corollary 1
  • Lemma 1
  • ...and 15 more