Diffusion-based Perceptual Neural Video Compression with Temporal Diffusion Information Reuse

TL;DR

This work targets perceptual video compression by integrating a foundational diffusion model into a conditional coding framework. DiffVC leverages temporal context from previously decoded frames and the current latent to guide diffusion-based reconstruction, and introduces Temporal Diffusion Information Reuse (TDIR) to cut inference time while maintaining perceptual quality. A Quantization Parameter-based Prompting (QPP) mechanism enables a single diffusion model to adapt across variable bitrates by modulating intermediate diffusion features via prompts. Training uses a multi-stage strategy to jointly optimize motion, contextual, and diffusion modules, with extensive experiments showing state-of-the-art perceptual metrics (notably DISTS) and substantial speedups, validating the approach for practical, bitrate-robust diffusion-based video compression.

Abstract

Recently, foundational diffusion models have attracted considerable attention in image compression tasks, whereas their application to video compression remains largely unexplored. In this article, we introduce DiffVC, a diffusion-based perceptual neural video compression framework that effectively integrates foundational diffusion model with the video conditional coding paradigm. This framework uses temporal context from previously decoded frame and the reconstructed latent representation of the current frame to guide the diffusion model in generating high-quality results. To accelerate the iterative inference process of diffusion model, we propose the Temporal Diffusion Information Reuse (TDIR) strategy, which significantly enhances inference efficiency with minimal performance loss by reusing the diffusion information from previous frames. Additionally, to address the challenges posed by distortion differences across various bitrates, we propose the Quantization Parameter-based Prompting (QPP) mechanism, which utilizes quantization parameters as prompts fed into the foundational diffusion model to explicitly modulate intermediate features, thereby enabling a robust variable bitrate diffusion-based neural compression framework. Experimental results demonstrate that our proposed solution delivers excellent performance in both perception metrics and visual quality.

Figures (9)

• Figure 1: The framework of DiffVC. DiffVC consists of three main components: Motion Modules, Contextual Modules, and Diffusion Modules. The Motion Modules (green) manage motion vector estimation and compression. The Contextual Modules (red) focus on extracting temporal context and compressing conditional residues. Finally, the Diffusion Modules (blue) apply multiple diffusion steps to generate high perceptual-quality reconstructions. The components $\mathcal{E}$ and $\mathcal{D}$ represent the pre-trained autoencoder of Stable Diffusion V2.1. The Frame and Feature Buffer stores the previous decoded frame and its latent representation feature, while the Diffusion Buffer stores diffusion information from the previous frame.
• Figure 2: Temporal Diffusion Information Reuse Strategy. The left panel illustrates the inference process of TDIR, where the vertical axis represents video frames and the horizontal axis represents diffusion timesteps. The first P frame undergoes independent diffusion for $DS$ steps, while subsequent P frames reuse diffuse for $DS-D$ steps before undergoing independent diffusion for the remaining $D$ steps. The right panel provides details on the independent diffusion step (red) and reuse diffusion step (green).
• Figure 3: QP-based Prompting Mechanism. The ratio of the quantization parameters, $q_{enc}$ and $q_{dec}$, in the Contextual Encoder/Decoder, is averaged channel-wise and encoded into tokens using the pretrained CLIP Text Encoder. These tokens are then used to modulate the intermediate features of the U-Net via cross-attention layers. The visualization in the top-right corner illustrates the relationship between the quantization parameter ratio and the bitrate, with the test data being the BQMall sequence from HEVC Class C.
• Figure 4: The rate-perception/distortion curves of our proposed DiffVC and other video compression methods on the HEVC dataset. Solid lines with dots represent traditional codecs, solid lines with triangles denote distortion-oriented neural video compression methods, dashed lines with circles indicate GAN-based neural video compression methods, and solid lines with circles correspond to Diffusion-based neural video compression methods.
• Figure 5: The rate-perception/distortion curves of our proposed DiffVC and other video compression methods on the MCL-JCV dataset. Solid lines with dots represent traditional codecs, solid lines with triangles denote distortion-oriented neural video compression methods, dashed lines with circles indicate GAN-based neural video compression methods, and solid lines with circles correspond to Diffusion-based neural video compression methods.