Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Communication Complexity of Exact Sampling under Rényi Information

Spencer Hill, Fady Alajaji, Tamás Linder

Abstract

We study the problem of exact sampling under an exponential communication cost, specifically Campbell's average codeword length $L(t)$ of order $t$, and Rényi's entropy. We provide a lower bound on the Campbell cost of exact sampling that grows approximately as $D_{1/α}(P||Q)$, the Rényi divergence of order $1/α$, with $α= \frac{1}{1+t}$. Using the Poisson functional representation of Li and El Gamal, we prove an upper bound on $L(t)$ whose leading Rényi divergence term has order within $ε$ of that of the lower bound. Our results reduce to the bounds of Harsha et al. as $α\to 1$. We also provide numerical examples comparing the bounds in the cases of normal and Laplacian distributions, demonstrating that the upper and lower bounds are typically within 5-10 bits of each other. Our results characterize exactly the optimal asymptotic Campbell cost $L(t)$ per sample as the number of independent and identically distributed (i.i.d.) samples grows to infinity. We show that under the exponential cost, any causal sampler performs strictly worse asymptotically than noncausal samplers. This contrasts with the case of expected message length, where both causal and noncausal samplers have the same optimal asymptotic cost.

Communication Complexity of Exact Sampling under Rényi Information

Abstract

We study the problem of exact sampling under an exponential communication cost, specifically Campbell's average codeword length of order , and Rényi's entropy. We provide a lower bound on the Campbell cost of exact sampling that grows approximately as , the Rényi divergence of order , with . Using the Poisson functional representation of Li and El Gamal, we prove an upper bound on whose leading Rényi divergence term has order within of that of the lower bound. Our results reduce to the bounds of Harsha et al. as . We also provide numerical examples comparing the bounds in the cases of normal and Laplacian distributions, demonstrating that the upper and lower bounds are typically within 5-10 bits of each other. Our results characterize exactly the optimal asymptotic Campbell cost per sample as the number of independent and identically distributed (i.i.d.) samples grows to infinity. We show that under the exponential cost, any causal sampler performs strictly worse asymptotically than noncausal samplers. This contrasts with the case of expected message length, where both causal and noncausal samplers have the same optimal asymptotic cost.

Paper Structure

This paper contains 16 sections, 13 theorems, 88 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Preliminaries and Contributions Summary
  3. Rényi Information Measures
  4. Problem Setup
  5. Prior Work
  6. Contributions
  7. Lower Bound on the Campbell Cost
  8. Upper Bounds using the Poisson Functional Representation
  9. Asymptotic Results
  10. Numerical Examples
  11. Conclusion
  12. Proof of Lemma \ref{['lemma:alphamomentlowerbound']}
  13. Proof of Proposition \ref{['prop:simpleconstant']}
  14. Proof of Lemma \ref{['lemma:geometricmoment']}
  15. Constant Term Analysis
  16. ...and 1 more sections

Key Result

Proposition 1

Suppose the discrete random variable $X$ is encoded using a uniquely decodable binary code $\mathcal{C} : \mathcal{X} \rightarrow \qty{0, 1}^*$ such that each codeword has length $l(\mathcal{C}(x)) = n_x$, for $x \in \mathcal{X}$. Then the average code length of order $t>0$ given by satisfies where $\alpha = \frac{1}{t + 1}$. Moreover, there exists a uniquely decodable binary code $\mathcal{C}$

Figures (6)

  • Figure 1: Block diagram of the exact sampling problem.
  • Figure 2: Block diagram of the channel simulation problem using sampling.
  • Figure 3: ${\min_{\epsilon > 0}} \; c_1(\alpha, \epsilon)$ and ${\min_{\epsilon > 0} \tilde{c}_1(\alpha, \epsilon)}$, as $\alpha$ varies between 0.2 and 1.
  • Figure 4: Comparison of the bounds on the Campbell cost for two normal distributions, for $0.2 < \alpha < 1$.
  • Figure 5: Comparison of the bounds on the Campbell cost and $H_\alpha(K)$ for $P =\mathcal{N}(0, 1)$ and $Q=\mathcal{N}(1, 1)$, for $0.2 < \alpha < 1$.
  • ...and 1 more figures

Theorems & Definitions (30)

  • Proposition 1: campbell1965coding
  • Definition 1: Poisson functional representation li2021unified
  • Theorem 1
  • Lemma 1
  • proof
  • proof : Proof of Theorem \ref{['thm:lowerbound']}
  • Theorem 2
  • Proposition 2
  • Lemma 2
  • Lemma 3
  • ...and 20 more