Prediction and Reference Quality Adaptation for Learned Video Compression

TL;DR

This work tackles prediction and reference quality adaptation in learned video compression by introducing a confidence-based Prediction Quality Adaptation (PQA) module and a Reference Quality Adaptation (RQA) module, along with a repeat-long training strategy. The PQA module generates spatial and channel-wise confidence maps to selectively exploit high-quality temporal contexts, while the RQA module provides dynamic, reference-aware filtering to curb reconstruction error propagation. Integrated into a DCVC-SDD-based framework with temporal context mining, a frame generator, a ConvLSTM long-term reference, and enhanced entropy modeling, the approach yields consistent bitrate reductions and improved reconstruction quality across RGB and YUV420 data, demonstrated through extensive ablations. The results indicate practical gains for learned video codecs, with modest increases in computation and memory, validating the effectiveness of explicit prediction- and reference-quality adaptation in temporal video processing.

Abstract

Temporal prediction is one of the most important technologies for video compression. Various prediction coding modes are designed in traditional video codecs. Traditional video codecs will adaptively to decide the optimal coding mode according to the prediction quality and reference quality. Recently, learned video codecs have made great progress. However, they did not effectively address the problem of prediction and reference quality adaptation, which limits the effective utilization of temporal prediction and reduction of reconstruction error propagation. Therefore, in this paper, we first propose a confidence-based prediction quality adaptation (PQA) module to provide explicit discrimination for the spatial and channel-wise prediction quality difference. With this module, the prediction with low quality will be suppressed and that with high quality will be enhanced. The codec can adaptively decide which spatial or channel location of predictions to use. Then, we further propose a reference quality adaptation (RQA) module and an associated repeat-long training strategy to provide dynamic spatially variant filters for diverse reference qualities. With these filters, our codec can adapt to different reference qualities, making it easier to achieve the target reconstruction quality and reduce the reconstruction error propagation. Experimental results verify that our proposed modules can effectively help our codec achieve a higher compression performance.

Figures (14)

• Figure 1: Overview of the learned video codec based on our proposed prediction quality adaptation module and reference quality adaptation module.
• Figure 2: Architecture of the contextual encoder-decoder and frame generator. The proposed prediction quality adaptation (PQA) module is used to provide explicit spatial and channel-wise discrimination for the prediction quality of temporal contexts. The proposed reference quality adaptation (RQA) module is used to dynamically adjust the contextual encoder-decoder to adapt to reference quality to reduce reconstruction error propagation.
• Figure 3: Visualization of the spatial and channel-wise prediction quality difference of temporal contexts.
• Figure 4: Architecture of our proposed prediction quality adaptation module.
• Figure 5: Illustration of the reconstruction quality difference across video frames of the reference software of H.266/VVC bross2021overview and our baseline---DCVC-SDD sheng2024spatial.