Practical Lossless Compression with Latent Variables using Bits Back Coding

TL;DR

This work presents BB-ANS, a practical lossless compression scheme for latent-variable models that uses bits back coding with ANS to achieve near-optimal rates. By connecting the coding length to the negative ELBO, the method enables efficient compression of data like MNIST using a VAE, including continuous latents via discretization. The authors demonstrate that BB-ANS outperforms standard baselines on MNIST and argue that state-of-the-art latent variable models could yield even larger gains, especially when parallelized on GPUs. They also discuss practical considerations such as discretization, 'clean' bits, and model communication, and provide open-source implementations. Overall, BB-ANS offers a pathway to high-efficiency lossless compression with modern generative models in scalable, parallelizable pipelines.

Abstract

Deep latent variable models have seen recent success in many data domains. Lossless compression is an application of these models which, despite having the potential to be highly useful, has yet to be implemented in a practical manner. We present `Bits Back with ANS' (BB-ANS), a scheme to perform lossless compression with latent variable models at a near optimal rate. We demonstrate this scheme by using it to compress the MNIST dataset with a variational auto-encoder model (VAE), achieving compression rates superior to standard methods with only a simple VAE. Given that the scheme is highly amenable to parallelization, we conclude that with a sufficiently high quality generative model this scheme could be used to achieve substantial improvements in compression rate with acceptable running time. We make our implementation available open source at https://github.com/bits-back/bits-back .

Figures (5)

• Figure 1: Visual comparison of 30 binarized MNIST images with bitstream outputs from running lossless compression algorithms PNG, bz2 and BB-ANS on the images.
• Figure 2: Graphical model with latent variable $y$ and observed variable $s$.
• Figure 3: A 2000 point moving average of the compression rate, in bits per dimension, during the compression process using BB-ANS with a VAE. We compress a concatenation of three shuffled copies of the MNIST test set.
• Figure 4: An example of the discretization of the latent space with a standard Gaussian prior, using 16 buckets.
• Figure 5: Python implementation of BB-ANS encode ('append') and decode ('pop') methods.