DiffVC-RT: Towards Practical Real-Time Diffusion-based Perceptual Neural Video Compression

TL;DR

DiffVC-RT tackles the practical barriers of diffusion-based perceptual neural video compression by delivering real-time performance through three coordinated innovations: an Efficient and Informative Model Architecture that replaces heavy encoders with PixelUnshuffle, expands latent channels, and uses pruned diffusion backbones; a Zero-Cost Consistency framework with an Online Temporal Shift Module and implicit temporal losses to stabilize frame-to-frame texture; and an Asynchronous and Parallel Decoding Pipeline with Batch-dimension Temporal Shift and Mixed Half Precision to maximize throughput. Empirically, it achieves substantial BD-Rate gains in perceptual metrics on HEVC datasets, while delivering real-time speeds (e.g., 206/30 fps at 720p on an NVIDIA H800) and significantly reduced model complexity compared with prior diffusion-based NVC methods. These advances bring diffusion-based perceptual NVC closer to industrial deployment, enabling high-quality, texture-rich video at practical bitrates and latencies. The work demonstrates a viable Pareto balance among perceptual quality, bitrate, and inference speed, highlighting DiffVC-RT as a practical path forward for diffusion-driven video compression.

Abstract

The practical deployment of diffusion-based Neural Video Compression (NVC) faces critical challenges, including severe information loss, prohibitive inference latency, and poor temporal consistency. To bridge this gap, we propose DiffVC-RT, the first framework designed to achieve real-time diffusion-based perceptual NVC. First, we introduce an Efficient and Informative Model Architecture. Through strategic module replacements and pruning, this architecture significantly reduces computational complexity while mitigating structural information loss. Second, to address generative flickering artifacts, we propose Explicit and Implicit Consistency Modeling. We enhance temporal consistency by explicitly incorporating a zero-cost Online Temporal Shift Module within the U-Net, complemented by hybrid implicit consistency constraints. Finally, we present an Asynchronous and Parallel Decoding Pipeline incorporating Mixed Half Precision, which enables asynchronous latent decoding and parallel frame reconstruction via a Batch-dimension Temporal Shift design. Experiments show that DiffVC-RT achieves 80.1% bitrate savings in terms of LPIPS over VTM-17.0 on HEVC dataset with real-time encoding and decoding speeds of 206 / 30 fps for 720p videos on an NVIDIA H800 GPU, marking a significant milestone in diffusion-based video compression.

Figures (10)

• Figure 1: Performance and efficiency overview of our proposed DiffVC-RT compared with various video codecs. The BD-Rate is measured using LPIPS on HEVC datasets, with VTM-17.0 serving as the anchor.
• Figure 2: The Efficient and Informative Model Architecture of DiffVC-RT.
• Figure 3: The Asynchronous and Parallel Decoding Pipeline.
• Figure 4: Rate-perception curves of the proposed DiffVC-RT and other methods on the HEVC, MCL-JCV, and UVG datasets.
• Figure 5: Visual comparison across different codecs.