Generative Latent Video Compression

TL;DR

GLVC tackles perceptual video compression by decoupling semantic fidelity from perceptual realism through a continuous latent space learned by a pretrained tokenizer. It introduces a latent video codec with unified intra-/inter-frame coding and a recurrent memory mechanism to preserve long-range temporal coherence, while perceptual details are synthesized by the tokenizer. On benchmarks, GLVC achieves state-of-the-art DISTS and LPIPS, and a user study shows competitive perceptual quality at roughly half the rate of leading neural or traditional codecs. Limitations include high encoding/decoding complexity and a four-frame latency due to temporal downsampling in the tokenizer.

Abstract

Perceptual optimization is widely recognized as essential for neural compression, yet balancing the rate-distortion-perception tradeoff remains challenging. This difficulty is especially pronounced in video compression, where frame-wise quality fluctuations often cause perceptually optimized neural video codecs to suffer from flickering artifacts. In this paper, inspired by the success of latent generative models, we present Generative Latent Video Compression (GLVC), an effective framework for perceptual video compression. GLVC employs a pretrained continuous tokenizer to project video frames into a perceptually aligned latent space, thereby offloading perceptual constraints from the rate-distortion optimization. We redesign the codec architecture explicitly for the latent domain, drawing on extensive insights from prior neural video codecs, and further equip it with innovations such as unified intra/inter coding and a recurrent memory mechanism. Experimental results across multiple benchmarks show that GLVC achieves state-of-the-art performance in terms of DISTS and LPIPS metrics. Notably, our user study confirms GLVC rivals the latest neural video codecs at nearly half their rate while maintaining stable temporal coherence, marking a step toward practical perceptual video compression.

Figures (12)

• Figure 1: Example of decoded videos. Click to play the video. Best viewed with Adobe Reader.
• Figure 2: (a) Visual generation process illustrated by diffusion models ho2020denoising. Information emerges first in the semantic regime, where bits allocated are strongly tied to distortion (MSE/PSNR). In the subsequent perceptual regime, a larger share of bits is consumed to enrich realism but contributing less to distortion. (b) GLVC performs rate–distortion optimization in a continuous semantic latent space while offloading perceptual detail synthesis to the tokenizer
• Figure 3: Our designed latent video compression module. It unifies both intra and inter compression and incorporates a recurrent memory mechanism that stores the historical semantic information from previously compressed latent representations.
• Figure 4: Rate-distortion curves in terms of DISTS and LPIPS.
• Figure 5: Visual comparisons of the decoded frames. Zoom in for best comparisons.