Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Discretized Approximate Ancestral Sampling

Alfredo De la Fuente, Saurabh Singh, Jona Ballé

TL;DR

The paper addresses the lack of an efficient sampling method for the Fourier Basis Density Model (FBM), a band-limited density that is cheap to evaluate. It introduces Discretized Approximate Ancestral Sampling (DAAS), which discretizes the FBM into an ancestor distribution over $K$ points and uses a nonnegative interpolation kernel to form a tractable $q(x)$ that approximates $p(x)$, with optional refinement via Unadjusted or Metropolis-adjusted Langevin dynamics. The authors prove rigorous bounds: $D_{TV}(p,q) \le C_1/K^2$ and $D_{W_1}(p,q) \le C_2/K^2$, and analyze computational complexity, showing near-linear scaling in the number of samples when using a triangular kernel. Empirical results demonstrate how increasing $K$ and incorporating MCMC refinements improves sample fidelity, while comparing against rejection sampling and highlighting trade-offs between accuracy and speed. This work enables scalable, controllable sampling from FBM, broadening its applicability in deep learning and Bayesian modeling.

Abstract

The Fourier Basis Density Model (FBM) was recently introduced as a flexible probability model for band-limited distributions, i.e. ones which are smooth in the sense of having a characteristic function with limited support around the origin. Its density and cumulative distribution functions can be efficiently evaluated and trained with stochastic optimization methods, which makes the model suitable for deep learning applications. However, the model lacked support for sampling. Here, we introduce a method inspired by discretization--interpolation methods common in Digital Signal Processing, which directly take advantage of the band-limited property. We review mathematical properties of the FBM, and prove quality bounds of the sampled distribution in terms of the total variation (TV) and Wasserstein--1 divergences from the model. These bounds can be used to inform the choice of hyperparameters to reach any desired sample quality. We discuss these results in comparison to a variety of other sampling techniques, highlighting tradeoffs between computational complexity and sampling quality.

Discretized Approximate Ancestral Sampling

TL;DR

The paper addresses the lack of an efficient sampling method for the Fourier Basis Density Model (FBM), a band-limited density that is cheap to evaluate. It introduces Discretized Approximate Ancestral Sampling (DAAS), which discretizes the FBM into an ancestor distribution over points and uses a nonnegative interpolation kernel to form a tractable that approximates , with optional refinement via Unadjusted or Metropolis-adjusted Langevin dynamics. The authors prove rigorous bounds: and , and analyze computational complexity, showing near-linear scaling in the number of samples when using a triangular kernel. Empirical results demonstrate how increasing and incorporating MCMC refinements improves sample fidelity, while comparing against rejection sampling and highlighting trade-offs between accuracy and speed. This work enables scalable, controllable sampling from FBM, broadening its applicability in deep learning and Bayesian modeling.

Abstract

The Fourier Basis Density Model (FBM) was recently introduced as a flexible probability model for band-limited distributions, i.e. ones which are smooth in the sense of having a characteristic function with limited support around the origin. Its density and cumulative distribution functions can be efficiently evaluated and trained with stochastic optimization methods, which makes the model suitable for deep learning applications. However, the model lacked support for sampling. Here, we introduce a method inspired by discretization--interpolation methods common in Digital Signal Processing, which directly take advantage of the band-limited property. We review mathematical properties of the FBM, and prove quality bounds of the sampled distribution in terms of the total variation (TV) and Wasserstein--1 divergences from the model. These bounds can be used to inform the choice of hyperparameters to reach any desired sample quality. We discuss these results in comparison to a variety of other sampling techniques, highlighting tradeoffs between computational complexity and sampling quality.
Paper Structure (27 sections, 4 theorems, 51 equations, 7 figures, 1 table, 1 algorithm)

This paper contains 27 sections, 4 theorems, 51 equations, 7 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Fourier Basis Density Model
  4. Discretized Approximate Ancestral Sampling
  5. Step 1: sampling from the ancestor
  6. Step 2: sampling from approximate conditional
  7. Step 3 (optional): refinement using MCMC
  8. Computational complexity
  9. Bounds for the approximation error
  10. Experimental evaluation
  11. Varying $K$
  12. Varying $T$
  13. Conclusion
  14. Positivity
  15. Proof of \ref{['prop:average_value']}
  16. ...and 12 more sections

Key Result

Proposition 1

Let $x_k = -1 + \frac{2k}{K}$ for $k=0, \dots, K-1$ be $K > 2N$ equally spaced points in the interval $[-1, 1)$. Then, $\sum_{k=0}^{K-1} p(x_k) = \frac{K}{2}$. For a proof, see Appendix sec:prop5_proof_app of the pre-print.

Figures (7)

  • Figure 1: Illustration of a band limited Fourier series (\ref{['eq:fourier_series']}) with only two frequency terms approximating a circular probability density within the range $[-1, 1)$.
  • Figure 2: Visual comparison of histograms obtained from Algorithm \ref{['alg:daas']} with the triangle kernel $w_1$ and varying $K =\{20, 50\}$ for an arbitrary multi-modal FBM density with $10$ frequency terms. In the case of the Nyquist rate ($K=20$), the histogram clearly illustrates the piecewise linear nature of $q(x)$ (light blue fill). As we increase $K$ to 50, $q(x)$ approximates the target distribution $p(x)$ more accurately (solid blue line).
  • Figure 3: Visualization of $D_{KL}$ decreasing w.r.t $K$ calculated between the unbiased samples from the target distribution via rejection sampling and the approximate samples obtained from our algorithm, considering $10$ randomly initialized FBMs. Left: We consider different number of frequencies $N=\{50, 100, 200\}$ for FBM initializations, and observe the same trend as $K$ grows. Right: We explore different B-spline kernels $w_D$ for $N=50$: the linear interpolation (triangular kernel $w_1$) performs significantly better empirically than the uniform ($w_0$) or piecewise quadratic spline ($w_2$).
  • Figure 4: Comparison of sampling methods in terms of Wasserstein--1 divergence $W_1$ against unbiased samples of $p(x)$ obtained via rejection sampling, for a randomly initialized FBM with $N=20$. We set $K = 4N$ and report results from ULA and MALA with optimized hyperparameters ($\epsilon_t = 10^{-5}$ and $\epsilon_t = 8 \times 10^{-5}$ respectively).
  • Figure 5: Visual comparison of empirical distributions by different sampling methods for an arbitrary multi-modal FBM density with $N=10$ frequency terms and $K=30$ sampling points. We observe how the empirical distributions of $10^6$ samples for each method differ to capture a few local minima/maxima present on the FBM distribution.
  • ...and 2 more figures

Theorems & Definitions (4)

  • Proposition 1: A.6, Gillman et al. gillman2024fourierheadhelpinglarge
  • Proposition 2
  • Theorem 1
  • Proposition 3