Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Don't Forget Its Variance! The Minimum Path Variance Principle for Accurate and Stable Score-Based Density Ratio Estimation

Wei Chen, Jiacheng Li, Shigui Li, Zhiqi Lin, Junmei Yang, John Paisley, Delu Zeng

TL;DR

This paper tackles the practical instability and path-dependence observed in score-based density ratio estimation (DRE), where theory asserts path-invariance but empirical performance hinges on the interpolation path. It identifies a missing path-variance term in the ideal objective and introduces the Minimum Path Variance (MinPV) principle, deriving closed-form expressions for the variance under common interpolants and enabling tractable optimization. To realize data-adaptive paths, the authors parameterize the path with a Kumaraswamy Mixture Model (KMM) and jointly optimize it with a time-score model using a CTSM-based objective, stabilized by a variance-aware time sampler. Empirical results across f-divergence estimation, MI, and density estimation benchmarks demonstrate state-of-the-art accuracy and stability, validating the effectiveness of learning the path geometry rather than fixing it a priori. The work has broad implications for improving reliability of score-based methods in density estimation and related probabilistic modeling tasks.

Abstract

Score-based methods have emerged as a powerful framework for density ratio estimation (DRE), but they face an important paradox in that, while theoretically path-independent, their practical performance depends critically on the chosen path schedule. We resolve this issue by proving that tractable training objectives differ from the ideal, ground-truth objective by a crucial, overlooked term: the path variance of the time score. To address this, we propose MinPV (\textbf{Min}imum \textbf{P}ath \textbf{V}ariance) Principle, which introduces a principled heuristic to minimize the overlooked path variance. Our key contribution is the derivation of a closed-form expression for the variance, turning an intractable problem into a tractable optimization. By parameterizing the path with a flexible Kumaraswamy Mixture Model, our method learns a data-adaptive, low-variance path without heuristic selection. This principled optimization of the complete objective yields more accurate and stable estimators, establishing new state-of-the-art results on challenging benchmarks.

Don't Forget Its Variance! The Minimum Path Variance Principle for Accurate and Stable Score-Based Density Ratio Estimation

TL;DR

This paper tackles the practical instability and path-dependence observed in score-based density ratio estimation (DRE), where theory asserts path-invariance but empirical performance hinges on the interpolation path. It identifies a missing path-variance term in the ideal objective and introduces the Minimum Path Variance (MinPV) principle, deriving closed-form expressions for the variance under common interpolants and enabling tractable optimization. To realize data-adaptive paths, the authors parameterize the path with a Kumaraswamy Mixture Model (KMM) and jointly optimize it with a time-score model using a CTSM-based objective, stabilized by a variance-aware time sampler. Empirical results across f-divergence estimation, MI, and density estimation benchmarks demonstrate state-of-the-art accuracy and stability, validating the effectiveness of learning the path geometry rather than fixing it a priori. The work has broad implications for improving reliability of score-based methods in density estimation and related probabilistic modeling tasks.

Abstract

Score-based methods have emerged as a powerful framework for density ratio estimation (DRE), but they face an important paradox in that, while theoretically path-independent, their practical performance depends critically on the chosen path schedule. We resolve this issue by proving that tractable training objectives differ from the ideal, ground-truth objective by a crucial, overlooked term: the path variance of the time score. To address this, we propose MinPV (\textbf{Min}imum \textbf{P}ath \textbf{V}ariance) Principle, which introduces a principled heuristic to minimize the overlooked path variance. Our key contribution is the derivation of a closed-form expression for the variance, turning an intractable problem into a tractable optimization. By parameterizing the path with a flexible Kumaraswamy Mixture Model, our method learns a data-adaptive, low-variance path without heuristic selection. This principled optimization of the complete objective yields more accurate and stable estimators, establishing new state-of-the-art results on challenging benchmarks.
Paper Structure (79 sections, 7 theorems, 70 equations, 10 figures, 6 tables, 3 algorithms)

This paper contains 79 sections, 7 theorems, 70 equations, 10 figures, 6 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Score-Based Density Ratio Estimation.
  4. The Importance of Paths in Probabilistic Inference and Generative Modeling.
  5. Bridging the Gap: Our Contribution.
  6. Background
  7. Interpolants.
  8. Density Ratio Estimation.
  9. Minimum Path Variance Principle
  10. The Path Variance: An Overlooked Term in the Objective
  11. Relationship to Existing Error Bounds.
  12. Remark on the Lipschitz Constant $L$ in \ref{['thm:principled-error-bound']}.
  13. Remark on Stabilizing $\mathcal{L}_{\text{STSM}}$ in \ref{['thm:principled-error-bound']}.
  14. Closed-form Path Variance.
  15. Minimizing the Path Variance via KMM Parameterization
  16. ...and 64 more sections

Key Result

Lemma 4.0

Let $\{p_t\}_{t\in[0,1]}$ be a probability path connecting $p_0$ and $p_1$. Under assumption:Lipschitz-continuous-logpt, the expected estimation error admits the following upper bound: where $\epsilon({\bm{x}}, t)=s^{(t)}({\bm{x}},t) - s_{{\bm{\theta}}}^{(t)}({\bm{x}},t)$ is the error of the score model, the exponents satisfy $\tfrac{1}{n} + \tfrac{1}{m} = 1$, and $L$ is the Lipschitz constant in

Figures (10)

  • Figure 1: (a) Preliminary experiments under various path settings. The left image denotes the ground truth. (b-c) Probability paths between $p_0$ and $p_1$ with large and small variance, respectively.
  • Figure 2: Density estimation on checkerboard and tree datasets. MinPV successfully learns a data-adaptive path tailored to the specific manifold of each dataset. Full results in \ref{['fig:toy2d_comparison']} of appendix.
  • Figure 3: Optimized KMM-parameterized path schedule for high-dimensional distributions.
  • Figure 4: Density estimation on six structured and multi-modal datasets. MinPV successfully learns a data-adaptive path tailored to the specific manifold of each dataset.
  • Figure 5: Optimized KMM-parameterized path schedule for geometrically pathological distributions.
  • ...and 5 more figures

Theorems & Definitions (13)

  • Lemma 4.0
  • Theorem 4.1: Minimum Path Variance Principle
  • Proposition 4.1
  • Lemma A.2
  • proof
  • Theorem A.2: Minimum Path Variance Principle
  • proof
  • Proposition A.2
  • proof
  • Remark A.3
  • ...and 3 more