Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SCORENF: Score-based Normalizing Flows for Sampling Unnormalized distributions

Vikas Kanaujia, Vipul Arora

TL;DR

ScoreNF introduces a score-based normalizing-flow framework with an IMH correction to sample from unnormalized distributions $p(\mathbf{x})$ efficiently. By jointly optimizing a score-matching term and a reverse-KL objective, ScoreNF mitigates both mode collapse and mode coverage shortcomings typical of traditional NF training, and it remains effective with small training ensembles. Empirical results on synthetic multimodal distributions (MOG-4, MOG-8) and a high-dimensional $\phi^4$ theory demonstrate superior density estimation (low NLL) and unbiased sampling (via IMH) compared to FKL, RKL, and SM baselines. The approach provides a data-efficient, scalable pathway for accurate inference on unnormalized targets with broad applicability in statistical physics and Bayesian computation.

Abstract

Unnormalized probability distributions are central to modeling complex physical systems across various scientific domains. Traditional sampling methods, such as Markov Chain Monte Carlo (MCMC), often suffer from slow convergence, critical slowing down, poor mode mixing, and high autocorrelation. In contrast, likelihood-based and adversarial machine learning models, though effective, are heavily data-driven, requiring large datasets and often encountering mode covering and mode collapse. In this work, we propose ScoreNF, a score-based learning framework built on the Normalizing Flow (NF) architecture, integrated with an Independent Metropolis-Hastings (IMH) module, enabling efficient and unbiased sampling from unnormalized target distributions. We show that ScoreNF maintains high performance even with small training ensembles, thereby reducing reliance on computationally expensive MCMC-generated training data. We also present a method for assessing mode-covering and mode-collapse behaviours. We validate our method on synthetic 2D distributions (MOG-4 and MOG-8) and the high-dimensional $φ^4$ lattice field theory distribution, demonstrating its effectiveness for sampling tasks.

SCORENF: Score-based Normalizing Flows for Sampling Unnormalized distributions

TL;DR

ScoreNF introduces a score-based normalizing-flow framework with an IMH correction to sample from unnormalized distributions efficiently. By jointly optimizing a score-matching term and a reverse-KL objective, ScoreNF mitigates both mode collapse and mode coverage shortcomings typical of traditional NF training, and it remains effective with small training ensembles. Empirical results on synthetic multimodal distributions (MOG-4, MOG-8) and a high-dimensional theory demonstrate superior density estimation (low NLL) and unbiased sampling (via IMH) compared to FKL, RKL, and SM baselines. The approach provides a data-efficient, scalable pathway for accurate inference on unnormalized targets with broad applicability in statistical physics and Bayesian computation.

Abstract

Unnormalized probability distributions are central to modeling complex physical systems across various scientific domains. Traditional sampling methods, such as Markov Chain Monte Carlo (MCMC), often suffer from slow convergence, critical slowing down, poor mode mixing, and high autocorrelation. In contrast, likelihood-based and adversarial machine learning models, though effective, are heavily data-driven, requiring large datasets and often encountering mode covering and mode collapse. In this work, we propose ScoreNF, a score-based learning framework built on the Normalizing Flow (NF) architecture, integrated with an Independent Metropolis-Hastings (IMH) module, enabling efficient and unbiased sampling from unnormalized target distributions. We show that ScoreNF maintains high performance even with small training ensembles, thereby reducing reliance on computationally expensive MCMC-generated training data. We also present a method for assessing mode-covering and mode-collapse behaviours. We validate our method on synthetic 2D distributions (MOG-4 and MOG-8) and the high-dimensional lattice field theory distribution, demonstrating its effectiveness for sampling tasks.
Paper Structure (12 sections, 13 equations, 1 figure, 4 tables)

This paper contains 12 sections, 13 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Problem Formulation
  4. Background
  5. Proposed Method
  6. Experiments
  7. Distributions
  8. Metrics
  9. Results and Discussion
  10. MOG-4 and MOG-8
  11. Scalar $\phi^4$ distribution
  12. Conclusions

Figures (1)

  • Figure 1: Sample plots for the MOG-4 and MOG-8 distributions generated using FKL, RKL, SM, and ScoreNF. Mode collapse is evident for the RKL approach, while mode covering behavior is observed in both FKL and SM approaches across all synthetic distributions.