Neural Video Compression with Feature Modulation

TL;DR

This work identifies two practical bottlenecks in neural video codecs based on conditional coding: a limited quality range and instability during long prediction chains. It introduces DCVC-FM, a feature-modulation-based codec built on DCVC-DC that employs a learnable quantization scaler and periodic temporal feature refreshing to widen the attainable PSNR range to about 11.4 dB and stabilize long-term predictions, while supporting a single RGB/YUV model and low-precision inference. The approach achieves substantial bitrate savings over both traditional codecs and prior SOTA neural codecs (e.g., up to 29.7% over DCVC-DC under intra-period --1) and demonstrates practical rate-control capabilities. These results advance the practicality of neural video compression, enabling finer quality control and cross-colorspace deployment with reduced computational cost, though real-time performance and cross-platform entropy-coding remain future challenges.

Abstract

The emerging conditional coding-based neural video codec (NVC) shows superiority over commonly-used residual coding-based codec and the latest NVC already claims to outperform the best traditional codec. However, there still exist critical problems blocking the practicality of NVC. In this paper, we propose a powerful conditional coding-based NVC that solves two critical problems via feature modulation. The first is how to support a wide quality range in a single model. Previous NVC with this capability only supports about 3.8 dB PSNR range on average. To tackle this limitation, we modulate the latent feature of the current frame via the learnable quantization scaler. During the training, we specially design the uniform quantization parameter sampling mechanism to improve the harmonization of encoding and quantization. This results in a better learning of the quantization scaler and helps our NVC support about 11.4 dB PSNR range. The second is how to make NVC still work under a long prediction chain. We expose that the previous SOTA NVC has an obvious quality degradation problem when using a large intra-period setting. To this end, we propose modulating the temporal feature with a periodically refreshing mechanism to boost the quality. %Besides solving the above two problems, we also design a single model that can support both RGB and YUV colorspaces. Notably, under single intra-frame setting, our codec can achieve 29.7\% bitrate saving over previous SOTA NVC with 16\% MACs reduction. Our codec serves as a notable landmark in the journey of NVC evolution. The codes are at https://github.com/microsoft/DCVC.

Figures (11)

• Figure 1: Rate-Distortion curve and BD-Rate comparisons with H.265/HM, H.266/VTM, ECM, and the previous SOTA NVC DCVC-DC li2023neural. The test dataset is HEVC E (600 frames) with single intra-frame setting (i.e. intra-period = --1) and YUV420 colorspace. DCVC-DC has a large performance drop under this setting while our new codec DCVC-FM still can significantly surpass ECM. Meanwhile, the quality range of our DCVC-FM is much larger than that of DCVC-DC.
• Figure 2: The framework of our DCVC-FM, built on DCVC-DC.
• Figure 3: The framework of our frame coding function $f_{frame}$. $E_{high}$ and $E_{low}$ are encoder at high and low resolution, respectively. $D_{high}$ and $D_{low}$ are corresponding decoder. AE and AD are arithmetic encoder and decoder. The quantization and inverse quantization processes are also applied to $f_{motion}$ in a similar way.
• Figure 4: Rate control examples using BasketballDrive video sequence (1080p, 50fps, 500 frames). The above (below) example is with a relatively high (low) target bitrate scenario.
• Figure 5: Temporal feature modulation via periodical refresh.