Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Complexity bounds for Dirichlet process slice samplers

Beatrice Franzolini, Francesco Gaffi

TL;DR

This paper analyzes the computational scalability of slice samplers for Dirichlet process (DP)–based models and establishes high-probability bounds on their per-iteration cost. By deriving a posteriori results, it shows that the dynamic truncation level exceeds the number of occupied clusters by at most a factor of order $\log n$ with high probability, ensuring that superlinear blow-ups occur only with vanishing probability. The approach leverages the DP stick-breaking representation and a refined posterior slice mechanism to bound the random number of instantiated components, independent of specific likelihood choices. Empirically, slice samplers exhibit favorable scalability relative to marginal CRP and fixed-truncation Gibbs methods while preserving exact posterior sampling. The findings provide a rigorous theoretical basis for the practical efficiency of DP slice-based inference across diverse datasets and growth regimes, with explicit dependence on the concentration parameter $\alpha$ and potential extensions to broader CRMs and hierarchical constructions.

Abstract

Slice sampling is a standard Monte Carlo technique for Dirichlet process (DP)-based models, widely used in posterior simulation. However, formal assessments of the scalability of posterior slice samplers have remained largely unexplored, primarily because the computational cost of a slice-sampling iteration is random and potentially unbounded. In this work, we obtain high-probability bounds on the computational complexity of DP slice samplers. Our main results show that, uniformly across posterior cluster-growth regimes, the overhead induced by slice variables, relatively to the number of clusters supported by the posterior, is $O_{\mathbb P}(\log n)$. As a consequence, even in worst-case configurations, superlinear blow-ups in per-iteration computational cost occur with vanishing probability. Our analysis applies broadly to DP-based models without any likelihood-specific assumptions, still providing complexity guarantees for posterior sampling on arbitrary datasets. These results establish a theoretical foundation for assessing the practical scalability of slice sampling in DP-based models.

Complexity bounds for Dirichlet process slice samplers

TL;DR

This paper analyzes the computational scalability of slice samplers for Dirichlet process (DP)–based models and establishes high-probability bounds on their per-iteration cost. By deriving a posteriori results, it shows that the dynamic truncation level exceeds the number of occupied clusters by at most a factor of order with high probability, ensuring that superlinear blow-ups occur only with vanishing probability. The approach leverages the DP stick-breaking representation and a refined posterior slice mechanism to bound the random number of instantiated components, independent of specific likelihood choices. Empirically, slice samplers exhibit favorable scalability relative to marginal CRP and fixed-truncation Gibbs methods while preserving exact posterior sampling. The findings provide a rigorous theoretical basis for the practical efficiency of DP slice-based inference across diverse datasets and growth regimes, with explicit dependence on the concentration parameter and potential extensions to broader CRMs and hierarchical constructions.

Abstract

Slice sampling is a standard Monte Carlo technique for Dirichlet process (DP)-based models, widely used in posterior simulation. However, formal assessments of the scalability of posterior slice samplers have remained largely unexplored, primarily because the computational cost of a slice-sampling iteration is random and potentially unbounded. In this work, we obtain high-probability bounds on the computational complexity of DP slice samplers. Our main results show that, uniformly across posterior cluster-growth regimes, the overhead induced by slice variables, relatively to the number of clusters supported by the posterior, is . As a consequence, even in worst-case configurations, superlinear blow-ups in per-iteration computational cost occur with vanishing probability. Our analysis applies broadly to DP-based models without any likelihood-specific assumptions, still providing complexity guarantees for posterior sampling on arbitrary datasets. These results establish a theoretical foundation for assessing the practical scalability of slice sampling in DP-based models.
Paper Structure (10 sections, 5 theorems, 75 equations, 11 figures, 1 table, 2 algorithms)

This paper contains 10 sections, 5 theorems, 75 equations, 11 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries on Dirichlet process and slice samplers
  3. High-probability bounds for slice samplers complexity
  4. Comparison with alternative algorithms for mixture models
  5. Numerical experiments
  6. Conclusion
  7. Proofs
  8. Additional details on numerical experiments
  9. Three equally-balanced clusters
  10. Perturbed Zipf

Key Result

Proposition 3.1

For any $\rho_n$ partition of $[n]$ with cluster sizes $(n_1,\dots,n_H)$, $\sum_{h=1}^H n_h=n$, let $(\pi_1,\dots,\pi_H,\pi^\star)\mid \rho_n \sim \mathrm{Dirichlet}(n_1,\dots,n_H,\alpha)$, let $u_i\mid \pi_{c_i}\sim \mathrm{Uniform}(0,\pi_{c_i})$ independently, and $u_{\min}=\min_{1\le i\le n}u_i$.

Figures (11)

  • Figure 1: Distributions involved in the slice sampler. Sampling $x\sim G(\text{d}x)$ is equivalent to sampling $u\sim p(u)$ and $x\mid u \sim p(x\mid u)$. Left panel: A realization of a discrete probability measure $G(\mathrm{d}x) = \sum_{k=1}^{\infty} \pi_k \delta_{\phi_k}(\mathrm{d}x)$, represented as point masses located at atoms $\phi_k$ with weights $\pi_k$. Sampling $x \sim G$ corresponds to selecting one atom $\phi_k$ with probability $\pi_k$. Center panel: The marginal distribution of the auxiliary slice variable $u$, given by $p(u) = \sum_{k\ge1} \mathbbm{1}(0<u<\pi_k)$, which is a decreasing step function supported on $(0, \max_k \pi_k)$. Right panel: Slice-sampling mechanism. First, a slice value $u_0$ is drawn from $p(u)$ (horizontal dashed line). Conditionally on $u_0$, the variable $x$ is sampled uniformly from the finite set $\{\phi_k : \pi_k > u_0\}$, corresponding to atoms whose weights exceed the slice level.
  • Figure 2: Wall-clock time per 1000 iterations in seconds computed as average over 10,000 iterations (including burn-in) as a function of input size (x-axis on log scale). Codes are in R and run on a laptop with 13th Gen Intel(R) Core(TM) i7-1370P.
  • Figure 3: ESS (w.r.t. to the log likelihood) per second computed as average over the last 5,000 iterations (excluding a burn-in of 5,000) as a function of input size (x-axis on log scale). Codes are in R and run on a laptop with 13th Gen Intel(R) Core(TM) i7-1370P.
  • Figure 4: Boxplots of the Rand index between the true clustering and the partitions visited by the MCMC algorithms, for each algorithm and sample size. Results are computed after a burn-in period of 5,000 iterations.
  • Figure 5: Three equally-balanced clusters: Rand index between the partition's point estimate and the true clustering configuration. The point estimate is obtained by minimizing the Binder loss function.
  • ...and 6 more figures

Theorems & Definitions (11)

  • Definition 2.1: DP--based generative model
  • Proposition 3.1: Merging two clusters increases the survival probability of $u_{\min}$
  • Corollary 3.2: Singleton partition yields the lowest survival probability of $u_{\min}$
  • Theorem 3.3: High‐probability bound on dynamic truncation level
  • Corollary 3.4
  • Corollary 3.5
  • proof : Proof of Proposition \ref{['prop:merge_two_clusters']}
  • proof : Proof of Corollary \ref{['cor:umin_singleton_worst']}
  • proof : Proof of Theorem \ref{['thm:bigtheorem']}
  • proof : Proof of Corollary \ref{['cor:exp_tail']}
  • ...and 1 more