Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Debiased Distribution Compression

Lingxiao Li, Raaz Dwivedi, Lester Mackey

TL;DR

This work tackles debiased distribution compression when only biased input samples are available, proposing a new suite of core algorithms that convert biased inputs into accurate summaries of a target distribution $\mathbb{P}$ with strong $\mathrm{MMD}$ guarantees. The core approach uses Stein kernels to debias input sequences and then compresses the debiased representation via Stein Kernel Thinning (SKT) to obtain $\sqrt{n}$ equal-weighted points with $\widetilde{O}(n^{-1/2})$ error, and extends this to scalable, sub-quadratic methods (Low-rank SKT). The framework further includes weighted variants—Simplex-weighted Stein Recombination and Stein Cholesky—that preserve simplex or constant constraints while maintaining comparable guarantees with $\operatorname{polylog}(n)$-sized coresets. The paper also provides new guarantees on the spectral decay of kernel matrices and covering numbers in Stein RKHS, and demonstrates practical effectiveness on posterior summaries in settings with burn-in, tempering, and approximate MCMC biases. Overall, the methods offer scalable, principled debiasing and compression that improve upon i.i.d. sampling in realistic Bayesian and MCMC contexts.

Abstract

Modern compression methods can summarize a target distribution $\mathbb{P}$ more succinctly than i.i.d. sampling but require access to a low-bias input sequence like a Markov chain converging quickly to $\mathbb{P}$. We introduce a new suite of compression methods suitable for compression with biased input sequences. Given $n$ points targeting the wrong distribution and quadratic time, Stein kernel thinning (SKT) returns $\sqrt{n}$ equal-weighted points with $\widetilde{O}(n^{-1/2})$ maximum mean discrepancy (MMD) to $\mathbb{P}$. For larger-scale compression tasks, low-rank SKT achieves the same feat in sub-quadratic time using an adaptive low-rank debiasing procedure that may be of independent interest. For downstream tasks that support simplex or constant-preserving weights, Stein recombination and Stein Cholesky achieve even greater parsimony, matching the guarantees of SKT with as few as $\text{poly-log}(n)$ weighted points. Underlying these advances are new guarantees for the quality of simplex-weighted coresets, the spectral decay of kernel matrices, and the covering numbers of Stein kernel Hilbert spaces. In our experiments, our techniques provide succinct and accurate posterior summaries while overcoming biases due to burn-in, approximate Markov chain Monte Carlo, and tempering.

Debiased Distribution Compression

TL;DR

This work tackles debiased distribution compression when only biased input samples are available, proposing a new suite of core algorithms that convert biased inputs into accurate summaries of a target distribution with strong guarantees. The core approach uses Stein kernels to debias input sequences and then compresses the debiased representation via Stein Kernel Thinning (SKT) to obtain equal-weighted points with error, and extends this to scalable, sub-quadratic methods (Low-rank SKT). The framework further includes weighted variants—Simplex-weighted Stein Recombination and Stein Cholesky—that preserve simplex or constant constraints while maintaining comparable guarantees with -sized coresets. The paper also provides new guarantees on the spectral decay of kernel matrices and covering numbers in Stein RKHS, and demonstrates practical effectiveness on posterior summaries in settings with burn-in, tempering, and approximate MCMC biases. Overall, the methods offer scalable, principled debiasing and compression that improve upon i.i.d. sampling in realistic Bayesian and MCMC contexts.

Abstract

Modern compression methods can summarize a target distribution more succinctly than i.i.d. sampling but require access to a low-bias input sequence like a Markov chain converging quickly to . We introduce a new suite of compression methods suitable for compression with biased input sequences. Given points targeting the wrong distribution and quadratic time, Stein kernel thinning (SKT) returns equal-weighted points with maximum mean discrepancy (MMD) to . For larger-scale compression tasks, low-rank SKT achieves the same feat in sub-quadratic time using an adaptive low-rank debiasing procedure that may be of independent interest. For downstream tasks that support simplex or constant-preserving weights, Stein recombination and Stein Cholesky achieve even greater parsimony, matching the guarantees of SKT with as few as weighted points. Underlying these advances are new guarantees for the quality of simplex-weighted coresets, the spectral decay of kernel matrices, and the covering numbers of Stein kernel Hilbert spaces. In our experiments, our techniques provide succinct and accurate posterior summaries while overcoming biases due to burn-in, approximate Markov chain Monte Carlo, and tempering.
Paper Structure (49 sections, 48 theorems, 281 equations, 6 figures, 2 tables, 18 algorithms)

This paper contains 49 sections, 48 theorems, 281 equations, 6 figures, 2 tables, 18 algorithms.

Table of Contents

  1. Introduction
  2. Debiased Distribution Compression
  3. Kernel assumptions
  4. Input point desiderata
  5. Debiased compression via Stein Kernel Thinning
  6. Accelerated Debiased Compression
  7. Fast bias correction via low-rank approximation
  8. Fast debiased compression via Low-rank Stein KT
  9. Weighted Debiased Compression
  10. Simplex-weighted coresets via Stein Recombination
  11. Constant-preserving coresets via Stein Cholesky
  12. Experiments
  13. Conclusions and Future Work
  14. Appendix Notation
  15. Spectral Analysis of Kernel Matrices
  16. ...and 34 more sections

Key Result

Proposition 1

A Stein kernel ${{\bm k}_{p}}$ with $\sup_{\|{x}\|_2 \le r}\|{\nabla \log p(x)}\|_2 = O(r^{d_{\ell}})$ for $d_{\ell}\geq 0$ is

Figures (6)

  • Figure 1: Correcting for burn-in.Left: Before selecting coresets (orange), the burn-in oracle uses 6 independent Markov chains to discard burn-in (red) while \ref{['alg:LSKT']} identifies the same high-density region (blue) with 1 chain. Right: Using only one chain, our methods consistently outperform the Stein and standard thinning baselines and match the 6-chain oracle.
  • Figure 2: Correcting for approximate MCMC (top) and tempering (bottom). For posterior inference over the parameters of Bayesian logistic regression ($d\!=\!54$, top) and a cardiac calcium signaling model ($d\!=\!38$, bottom), our concise coreset constructions correct for approximate MCMC and tempering biases without need for explicit importance sampling.
  • Figure 9.1: Correcting for burn-in with equal-weighted compression. For each of four MCMC algorithms and using only one chain, our methods consistently outperform the Stein and standard thinning baselines and match the 6-chain oracle.
  • Figure 9.2: Correcting for burn-in with simplex-weighted compression. For each of four MCMC algorithms and using only one chain, our methods consistently outperform the Stein and standard thinning baselines and match the 6-chain oracle.
  • Figure 9.3: Correcting for burn-in with constant-preserving compression. For each of four MCMC algorithms and using only one chain, our methods consistently outperform the Stein and standard thinning baselines and match the 6-chain oracle.
  • ...and 1 more figures

Theorems & Definitions (105)

  • Definition 1: Stein kernel
  • Definition 2: Covering number
  • Proposition 1: Stein kernel growth rates
  • Theorem 1: Debiasing to i.i.d. quality via simplex reweighting
  • Remark 1
  • Theorem 2: Better-than-i.i.d. debiasing via simplex reweighting
  • Definition 3: Slow-growing input points
  • Theorem 3: MMD guarantee for \ref{['alg:GSKT']}
  • Example 1
  • Remark 2
  • ...and 95 more