Compression-Realized Deep Structural Network for Video Quality Enhancement

TL;DR

The paper addresses how to improve video quality after compression by introducing codec-informed inductive biases into a deep VQE model. It proposes a Compression-Realized Deep Structural Network (CRDS) consisting of Mutual Neighborhood Attention (MuNA), Latent Degradation Residual Auto-Encoder (LDR-AE), and Progressive Denoising with Intermediate Supervision (PDIS), which operate in a latent space aligned with classic codec stages. By modeling residuals in latent space, progressively denoising them, and guiding motion estimation with mutual neighborhood attention, CRDS achieves state-of-the-art results on LDV 2.0 and MFQE 2.0 and shows strong out-of-distribution generalization. The work demonstrates that integrating compression priors into network design yields robust, interpretable VQE with practical implications for streaming and storage efficiency.

Abstract

This paper focuses on the task of quality enhancement for compressed videos. Although deep network-based video restorers achieve impressive progress, most of the existing methods lack a structured design to optimally leverage the priors within compression codecs. Since the quality degradation of the video is primarily induced by the compression algorithm, a new paradigm is urgently needed for a more ``conscious'' process of quality enhancement. As a result, we propose the Compression-Realized Deep Structural Network (CRDS), introducing three inductive biases aligned with the three primary processes in the classic compression codec, merging the strengths of classical encoder architecture with deep network capabilities. Inspired by the residual extraction and domain transformation process in the codec, a pre-trained Latent Degradation Residual Auto-Encoder is proposed to transform video frames into a latent feature space, and the mutual neighborhood attention mechanism is integrated for precise motion estimation and residual extraction. Furthermore, drawing inspiration from the quantization noise distribution of the codec, CRDS proposes a novel Progressive Denoising framework with intermediate supervision that decomposes the quality enhancement into a series of simpler denoising sub-tasks. Experimental results on datasets like LDV 2.0 and MFQE 2.0 indicate our approach surpasses state-of-the-art models.

Figures (5)

• Figure 1: The comparison between the Codec Compression framework and the CRDS framework.
• Figure 2: An Overview of our proposed CRDS framework. The CRDS framework comprises three primary structures: MuNA, LDR-AE, and PDIS, which adequately introduced the inductive bias of video compression.
• Figure 3: A visual demonstration of Mutual Neighborhood Attention. MuNA-ME estimates motion vectors by performing mutual attention on two frames. MuNA-AGP carries out feature warping using attention weights to produce the predicted frame.
• Figure 4: Qualitative comparison between CRDS and existing SOTA baselines on LDV 2.0 yang2022ntire.
• Figure 5: Visualization of motion vectors and residuals from x264 x264 codec and CRDS, where obvious similarity can be observed.