Some Notes on the Sample Complexity of Approximate Channel Simulation
Gergely Flamich, Lennie Wells
TL;DR
The paper analyzes the sample complexity of approximate one-shot channel simulation (OSCS). It proves a super-polynomial runtime lower bound in $D_{\infty}[Q||P]$ for general approximate schemes, implying hardness of approximate sampling in the worst case. It then identifies a favorable regime: with access to the unnormalized density ratio $r \propto dQ/dP$ and knowledge of $D_{KL}[Q||P]$, depth-limited A* coding yields $D_{TV}[Q||P] \le \epsilon$ with sample complexity $\exp_2\big((D_{KL}[Q||P] + o(1))/\epsilon\big)$, enabling near-optimal encoding rates. To push practical applicability, the authors refine the rejection-sampling approach via a better target $Q_M$ and show that depth-limited A* (or equivalent) can achieve similar TV guarantees with favorable dependence on the $f$-divergence and $D_{KL}$, avoiding dependence on $\|r\|_\infty$. Finally, they adapt these ideas to OSCS by replacing rejection sampling with depth-limited A* coding, achieving near-optimal encoding lengths and providing a roadmap for realism-constrained lossy compression pipelines that leverage structure in density ratios and KL divergence.
Abstract
Channel simulation algorithms can efficiently encode random samples from a prescribed target distribution $Q$ and find applications in machine learning-based lossy data compression. However, algorithms that encode exact samples usually have random runtime, limiting their applicability when a consistent encoding time is desirable. Thus, this paper considers approximate schemes with a fixed runtime instead. First, we strengthen a result of Agustsson and Theis and show that there is a class of pairs of target distribution $Q$ and coding distribution $P$, for which the runtime of any approximate scheme scales at least super-polynomially in $D_\infty[Q \Vert P]$. We then show, by contrast, that if we have access to an unnormalised Radon-Nikodym derivative $r \propto dQ/dP$ and knowledge of $D_{KL}[Q \Vert P]$, we can exploit global-bound, depth-limited A* coding to ensure $\mathrm{TV}[Q \Vert P] \leq ε$ and maintain optimal coding performance with a sample complexity of only $\exp_2\big((D_{KL}[Q \Vert P] + o(1)) \big/ ε\big)$.