Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Channel Simulation with Causal Rejection Samplers

Daniel Goc, Gergely Flamich

TL;DR

The paper addresses OSCS by focusing on a crucial CRS subclass, establishing a lower bound on CRS runtime of $\mathbb{E}[K] \ge \exp_2\big(D_{\infty}[Q||P]\big)$ and introducing the channel simulation divergence $D_{CS}[Q||P]$ to bound codelength via $D_{KL}[Q||P] \le D_{CS}[Q||P] \le \mathbb{H}[K]$. It proves a tight lower bound on coding length and demonstrates achievability with Greedy Rejection Sampling: $\mathbb{H}[K] \le D_{CS}[Q||P] + \log_2(e+1)$. The work further develops the theory by showing convexity properties and integral representations of $D_{CS}$, and provides numerical studies for Laplace and Gaussian channels to illustrate the tightness and limitations of KL-based bounds in practice. These results offer a precise, actionable characterization of one-shot channel simulation limits and guide design of efficient CRS-based schemes.

Abstract

One-shot channel simulation has recently emerged as a promising alternative to quantization and entropy coding in machine-learning-based lossy data compression schemes. However, while there are several potential applications of channel simulation - lossy compression with realism constraints or differential privacy, to name a few - little is known about its fundamental limitations. In this paper, we restrict our attention to a subclass of channel simulation protocols called causal rejection samplers (CRS), establish new, tighter lower bounds on their expected runtime and codelength, and demonstrate the bounds' achievability. Concretely, for an arbitrary CRS, let $Q$ and $P$ denote a target and proposal distribution supplied as input, and let $K$ be the number of samples examined by the algorithm. We show that the expected runtime $\mathbb{E}[K]$ of any CRS scales at least as $\exp_2(D_\infty[Q || P])$, where $D_\infty[Q || P]$ is the Rényi $\infty$-divergence. Regarding the codelength, we show that $D_{KL}[Q || P] \leq D_{CS}[Q || P] \leq \mathbb{H}[K]$, where $D_{CS}[Q || P]$ is a new quantity we call the channel simulation divergence. Furthermore, we prove that our new lower bound, unlike the $D_{KL}[Q || P]$ lower bound, is achievable tightly, i.e. there is a CRS such that $\mathbb{H}[K] \leq D_{CS}[Q || P] + \log_2 (e + 1)$. Finally, we conduct numerical studies of the asymptotic scaling of the codelength of Gaussian and Laplace channel simulation algorithms.

On Channel Simulation with Causal Rejection Samplers

TL;DR

The paper addresses OSCS by focusing on a crucial CRS subclass, establishing a lower bound on CRS runtime of and introducing the channel simulation divergence to bound codelength via . It proves a tight lower bound on coding length and demonstrates achievability with Greedy Rejection Sampling: . The work further develops the theory by showing convexity properties and integral representations of , and provides numerical studies for Laplace and Gaussian channels to illustrate the tightness and limitations of KL-based bounds in practice. These results offer a precise, actionable characterization of one-shot channel simulation limits and guide design of efficient CRS-based schemes.

Abstract

One-shot channel simulation has recently emerged as a promising alternative to quantization and entropy coding in machine-learning-based lossy data compression schemes. However, while there are several potential applications of channel simulation - lossy compression with realism constraints or differential privacy, to name a few - little is known about its fundamental limitations. In this paper, we restrict our attention to a subclass of channel simulation protocols called causal rejection samplers (CRS), establish new, tighter lower bounds on their expected runtime and codelength, and demonstrate the bounds' achievability. Concretely, for an arbitrary CRS, let and denote a target and proposal distribution supplied as input, and let be the number of samples examined by the algorithm. We show that the expected runtime of any CRS scales at least as , where is the Rényi -divergence. Regarding the codelength, we show that , where is a new quantity we call the channel simulation divergence. Furthermore, we prove that our new lower bound, unlike the lower bound, is achievable tightly, i.e. there is a CRS such that . Finally, we conduct numerical studies of the asymptotic scaling of the codelength of Gaussian and Laplace channel simulation algorithms.
Paper Structure (13 sections, 14 theorems, 90 equations, 1 figure, 1 algorithm)

This paper contains 13 sections, 14 theorems, 90 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Runtime Bound for Causal Rejection Samplers
  4. The Channel Simulation Divergence
  5. The Codelength of Causal Rejection Samplers
  6. Laplace and Gaussian Channel Simulation
  7. Discussion and Future Work
  8. Expanded Definitions
  9. The Connection Between the Channel Simulation Divergence, Excess Functional Information and Layered Entropy
  10. Convexity of Channel Simulation Divergence
  11. Upper Bounds on (Alternative) Channel Simulation Divergence
  12. Extended Causal Rejection Samplers
  13. Additional Results for the Greedy Rejection Sampler

Key Result

Theorem 3.1

Let $K$ be an index returned by a CRS with proposal $P$ and target $Q$. Then,

Figures (1)

  • Figure 1: Numerical demonstration of the looseness of the channel simulation bound in \ref{['eq:joint_sandwich_bound']}. (A) We plot $\Delta(Q, P)$ for $Q = \mathcal{L}(0, b)$ and $P = \mathcal{L}(0, 1)$ as a function of the target log-precision $-\ln b$. (B) We plot $\Delta(Q, P)$ for $Q = \mathcal{N}(1, 1/4)^{\otimes d}$ and $P = \mathcal{N}(0, 1)^{\otimes d}$ as a function of the dimension $d$.

Theorems & Definitions (29)

  • Definition 2.1: Causal Rejection Sampler
  • Theorem 3.1
  • proof
  • Definition 4.1: Channel Simulation Divergence
  • Lemma 4.1
  • proof
  • Theorem 5.1
  • proof
  • Lemma 5.1
  • proof
  • ...and 19 more