High-Efficiency Neural Video Compression via Hierarchical Predictive Learning

TL;DR

The enhanced Deep Hierarchical Video Compression-DHVC 2.0-has been introduced, delivering not only superior compression performance to representative methods but also impressive complexity efficiency, enabling real-time processing with a significantly smaller memory footprint on standard GPUs.

Abstract

The enhanced Deep Hierarchical Video Compression-DHVC 2.0-has been introduced. This single-model neural video codec operates across a broad range of bitrates, delivering not only superior compression performance to representative methods but also impressive complexity efficiency, enabling real-time processing with a significantly smaller memory footprint on standard GPUs. These remarkable advancements stem from the use of hierarchical predictive coding. Each video frame is uniformly transformed into multiscale representations through hierarchical variational autoencoders. For a specific scale's feature representation of a frame, its corresponding latent residual variables are generated by referencing lower-scale spatial features from the same frame and then conditionally entropy-encoded using a probabilistic model whose parameters are predicted using same-scale temporal reference from previous frames and lower-scale spatial reference of the current frame. This feature-space processing operates from the lowest to the highest scale of each frame, completely eliminating the need for the complexity-intensive motion estimation and compensation techniques that have been standard in video codecs for decades. The hierarchical approach facilitates parallel processing, accelerating both encoding and decoding, and supports transmission-friendly progressive decoding, making it particularly advantageous for networked video applications in the presence of packet loss. Source codes will be made available.

Figures (22)

• Figure 1: Rate-distortion-complexity of representative learned video codecs. The horizontal axis represents the decoding speed in frames per second (FPS), and the vertical axis shows the BD-rate bjontegaard2001calculation relative to HEVC reference model HM 16.26. The marker's size is proportional to the running memory of each method. The proposed DHVC 2.0 demonstrates a well-justified performance-complexity balance using a single model (marked with grid pattern). Particularly, DHVC 2.0 (Parallel) is the only one running in real-time (i.e., over 25 FPS) with superior performance to HM16.26.
• Figure 2: Inter-frame coding using (a) hybrid motion & residual coding lu2019dvcli2021deep, (b) single-scale predictive coding liu2020conditionalmentzer2022vct, and (c) the proposed hierarchical predictive coding lu2024deep. $x$: pixel-space image; $v$: motion flow; $z$: feature-space latent variable; $Z, M$: accumulated reference features for conditional coding.
• Figure 3: Visualizations of the distribution of latent variables for representative learned compression methods. (a) image coder using Joint minnen2018joint, and video coders using respective (b) DVC lu2019dvc, (c) DCVC-DC Li_2023_CVPR, (d) VCT mentzer2022vct, and (e) DHVC (hierarchical scale 1 to 4 from the leftmost to rightmost subplot). Joint and VCT share a similar distribution without implementing a residual coding mechanism as others. DHVC performs scale-wise residual coding from scale 1 (the leftmost subplot) to scale 4 (the rightmost subplot).
• Figure 4: A sketch of hierarchical predictive coding framework used by DHVC. As scale-wise processing is repeated, we only exemplify the $l$-th scale. "Down $\downarrow$" & "Up $\uparrow$" stand for down- & up-sampling, and "Concat." represents feature-space concatenation. Neural networks are stacked to realize these operations.
• Figure 5: Latent Block for conditional coding at each scale, which has thee modes: (a) training, (b) compressing, and (c) decompressing. In training, our model learns to minimize the KL divergence between all pairs of priors and posteriors, as well as the distortion between $\hat{x}_t$ and $x_t$. In compression inference, the uniform posteriors are replaced by uniform quantization, and the priors are turned into discrete PMFs (probability mass functions) for entropy coding.