Quantised Global Autoencoder: A Holistic Approach to Representing Visual Data

TL;DR

This work proposes a Quantised Global Autoencoder (QG-VAE) that replaces local patch-based tokens with globally informed tokens, enabling a holistic representation of images. It learns a set of pseudo-frequency components via a shared transform applied to multiple feature maps, with per-channel codebooks and a Voronoi-inspired reset mechanism to balance information across tokens. A Laplacian-pyramid-inspired regularisation orders the latent space, yielding interpretable progressive reconstructions, while a QGGAN extension demonstrates sharpening and improved generation. The approach delivers strong compression performance, sharper imagery, and a flexible latent space that supports downstream tasks, suggesting a new direction for autoencoders that prioritise global structure over local patches.

Abstract

In quantised autoencoders, images are usually split into local patches, each encoded by one token. This representation is redundant in the sense that the same number of tokens is spend per region, regardless of the visual information content in that region. Adaptive discretisation schemes like quadtrees are applied to allocate tokens for patches with varying sizes, but this just varies the region of influence for a token which nevertheless remains a local descriptor. Modern architectures add an attention mechanism to the autoencoder which infuses some degree of global information into the local tokens. Despite the global context, tokens are still associated with a local image region. In contrast, our method is inspired by spectral decompositions which transform an input signal into a superposition of global frequencies. Taking the data-driven perspective, we learn custom basis functions corresponding to the codebook entries in our VQ-VAE setup. Furthermore, a decoder combines these basis functions in a non-linear fashion, going beyond the simple linear superposition of spectral decompositions. We can achieve this global description with an efficient transpose operation between features and channels and demonstrate our performance on compression.

Figures (23)

• Figure 1: Our autoencoder produces a number of global tokens that represent the image, with each token influencing the coloured region of the image. These global tokens help to spread the unevenly distributed information content in the image across the tokens. Example shows $256$ tokens, a ratio of $64$ pixels to one token in $[1,512]$, i.e.$288$ byte for an image from ImageNetimagenet.
• Figure 2: Our base architecture: We encode our data by increasing the input to $C$ different feature maps in the channels (desired number of codewords) with a U-Net unet, then compress each channel individually into a token. We do so by transposing channel and collapsed feature dimension, then applying a single linear layer / affine transformation $K$, split into different heads (green and violet). For quantisation, we use the same idea as a naive VQ-VAE. We then decode the result in the same manner.
• Figure 3: We express re-initialising codewords while relieving overused codewords as re-arranging a Voronoi diagram through moving the Voronoi centres.
• Figure 4: Input, output, and absolute difference between the two, produced by our autoencoder. $256$ tokens/$288$ byte for ImageNetimagenet and CelebAceleba, $64$ tokens/$72$ bytes for CIFAR-10cifar. All with $512$ tokens in the codebook. Note how sharp the features, e.g. the horse rider, in CIFAR are.
• Figure 5: Examples on ImageNet $128$ with sharpening, from left to right: Input, output (QG-VAE), output (QGGAN). Each image has only $256$ tokens ($288$ bytes).