Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Rejection Sampling in the Entropy-Optimal Range

Thomas L. Draper, Feras A. Saad

TL;DR

This paper addresses exact sampling from finite discrete distributions using unbiased coin flips by introducing the Amplified Loaded Dice Roller (ALDR), a family of rejection samplers that blends Knuth–Yao entropy-optimal ideas with rejection sampling. ALDR achieves a near-entropy-optimal cost, with an expected entropy cost bound of $H(P) \le \mathbb{E}[\mathscr{C}(ALDR)] < H(P) + 2$ for depth parameter $K \ge 2k$, while preserving linearithmic space $O(n\log m\log n)$ and practical preprocessing. The work also provides a detailed analysis of ALDR’s toll, tight bounds for $K=2k$, and comparisons to FLDR and the Alias method, supplemented by implementable integer-arithmetic algorithms and numerical results showing runtime and entropy improvements. Overall, the results offer a scalable, exact sampling approach with favorable entropy efficiency suitable for hardware-constrained or entropy-sensitive environments, advancing discrete-distribution sampling beyond the Alias method. The paper also identifies limitations, such as potential gaps where ALDR is not entropy-optimal and questions around optimal amplification strategies.

Abstract

The problem of generating a random variate $X$ from a finite discrete probability distribution $P$ using an entropy source of independent unbiased coin flips is considered. The Knuth and Yao complexity theory of nonuniform random number generation furnishes a family of "entropy-optimal" sampling algorithms that consume between $H(P)$ and $H(P)+2$ coin flips per generated output, where $H$ is the Shannon entropy function. However, the space complexity of entropy-optimal samplers scales exponentially with the number of bits required to encode $P$. This article introduces a family of efficient rejection samplers and characterizes their entropy, space, and time complexity. Within this family is a distinguished sampling algorithm that requires linearithmic space and preprocessing time, and whose expected entropy cost always falls in the entropy-optimal range $[H(P), H(P)+2)$. No previous sampler for discrete probability distributions is known to achieve these characteristics. Numerical experiments demonstrate performance improvements in runtime and entropy of the proposed algorithm compared to the celebrated alias method.

Efficient Rejection Sampling in the Entropy-Optimal Range

TL;DR

This paper addresses exact sampling from finite discrete distributions using unbiased coin flips by introducing the Amplified Loaded Dice Roller (ALDR), a family of rejection samplers that blends Knuth–Yao entropy-optimal ideas with rejection sampling. ALDR achieves a near-entropy-optimal cost, with an expected entropy cost bound of for depth parameter , while preserving linearithmic space and practical preprocessing. The work also provides a detailed analysis of ALDR’s toll, tight bounds for , and comparisons to FLDR and the Alias method, supplemented by implementable integer-arithmetic algorithms and numerical results showing runtime and entropy improvements. Overall, the results offer a scalable, exact sampling approach with favorable entropy efficiency suitable for hardware-constrained or entropy-sensitive environments, advancing discrete-distribution sampling beyond the Alias method. The paper also identifies limitations, such as potential gaps where ALDR is not entropy-optimal and questions around optimal amplification strategies.

Abstract

The problem of generating a random variate from a finite discrete probability distribution using an entropy source of independent unbiased coin flips is considered. The Knuth and Yao complexity theory of nonuniform random number generation furnishes a family of "entropy-optimal" sampling algorithms that consume between and coin flips per generated output, where is the Shannon entropy function. However, the space complexity of entropy-optimal samplers scales exponentially with the number of bits required to encode . This article introduces a family of efficient rejection samplers and characterizes their entropy, space, and time complexity. Within this family is a distinguished sampling algorithm that requires linearithmic space and preprocessing time, and whose expected entropy cost always falls in the entropy-optimal range . No previous sampler for discrete probability distributions is known to achieve these characteristics. Numerical experiments demonstrate performance improvements in runtime and entropy of the proposed algorithm compared to the celebrated alias method.

Paper Structure

This paper contains 20 sections, 24 theorems, 58 equations, 10 figures, 3 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Main Result
  3. Related Sampling Techniques
  4. Overview of Theorems
  5. Preliminaries
  6. Discrete Distribution Generating Trees
  7. Entropy-Optimal DDG Trees
  8. Rejection Sampling
  9. Fast Loaded Dice Roller
  10. Amplified Loaded Dice Roller
  11. Main Idea
  12. Analysis of Expected Entropy Cost
  13. Tighter Bound on Entropy Cost for K=2k
  14. Further Entropy Properties
  15. Comparison of ALDR and Entropy-Optimal Sampling
  16. ...and 5 more sections

Key Result

theorem 1

Let $P \coloneqq (p_1, \dots, p_n)$ denote a discrete probability distribution over $n$ outcomes. Let $\mathcal{T}(P) = \lbrace T : \lbrace0,1\rbrace^{*} \rightharpoonup \mathbb{N} \mid P_T = P\rbrace$ denote the set of all DDG trees whose output distribution is $P$. The following statements regardi

Figures (10)

  • Figure 1: DDG tree representations of four random sampling algorithms $T:\lbrace0,1\rbrace^* \rightharpoonup \mathbb{N}$ with output distributions $P$. These trees are constructed using the entropy-optimal knuth1976 method from \ref{['theorem:knuth-yao']}. The string ${\mathbf{b}} \in \lbrace0,1\rbrace^*$ ranges over all finite-length bit string continuations. Any string ${\mathbf{b}}$ that does not index a path to a leaf node is not in the domain of $T$. The arrows and labels along the edges are omitted from DDG trees going forward. Leaves will also be labeled $a_i$ instead of $i$ for clarity when there are no duplicate weights.
  • Figure 2: Comparison of DDG trees for target distribution $P = (1/5, 4/5)$ with FLDR proposal ${Q} = (1/8, 4/8, 3/8)$
  • Figure 3: FLDR trees with tolls rapidly approaching $6$ bits.
  • Figure 4: DDG trees and entropy tolls of $\textrm{ALDR}$ samplers for $P = (4,7,8)/19$ and various tree depths $K$. The $\textrm{ALDR}$ sampler coincides with the $\textrm{FLDR}$ sampler, while $\textrm{ALDR}$ coincides with an entropy-optimal sampler $\textrm{KY}(P)$ from \ref{['theorem:knuth-yao']}.
  • Figure 5: Upper bound on the entropy cost of the ALDR sampler with increasing depth $K$, using the generic bound $\tau(\textrm{KY}) < 2$. The solid line shows a bound in terms of the binary entropy function \ref{['eq:aldr-generic-actual']} and the dashed line shows a simple upper bound on this quantity following \ref{['theorem:aldr-generic-bound']}, all as a function of $\delta \coloneqq K-k$.
  • ...and 5 more figures

Theorems & Definitions (64)

  • definition 1: han1993, Definition 4
  • theorem 1: knuth1976
  • theorem 2: knuth1976
  • proof : Proof (Sketch)
  • remark 1
  • remark 2
  • theorem 3: saad2020fldr
  • proof
  • proposition 1: topsoe2001
  • theorem 4: Tightness of FLDR toll bound
  • ...and 54 more