Decomposition, Compression, and Synthesis (DCS)-based Video Coding: A Neural Exploration via Resolution-Adaptive Learning

TL;DR

This work introduces Decomposition, Compression, and Synthesis (DCS), a codec-agnostic framework that mimics retinal processing by splitting video into STFs at native resolution and TMFs at lowered resolution for separate encoding. A resolution-adaptive synthesis module combines a multi-frame motion compensation network (MCN) and a non-local texture transfer network (NL-TTN) to restore high-resolution details and temporal continuity, expressed as $O^{H}_{t} = \mathcal{G}(\mathcal{F}(\{\hat{I}^{L}_{t-N}, ..., \hat{I}^{L}_{t}, ..., \hat{I}^{L}_{t+N}\}))$ with $F^{L}_{t} = \Phi(\Psi(\{\hat{I}^{L}_{t-N}, ..., \hat{I}^{L}_{t}, ..., \hat{I}^{L}_{t+N}\}))$. Trained end-to-end on a mixed dataset and evaluated against HEVC HM, DCS yields ~1 dB PSNR gains or ~25% BD-rate savings on average, confirming robust RD improvements and suggesting codec-agnostic applicability. The approach leverages bicubic resampling for TMFs and HEVC for compression, while ensuring compatibility with common standards and delivering notable gains across content types and GoP settings. The combination of deformable convolution–based alignment, temporal-spatial attention, and semantically guided texture transfer enables efficient reconstruction with controlled complexity, opening a practical path toward next-generation video coding.

Abstract

Inspired by the facts that retinal cells actually segregate the visual scene into different attributes (e.g., spatial details, temporal motion) for respective neuronal processing, we propose to first decompose the input video into respective spatial texture frames (STF) at its native spatial resolution that preserve the rich spatial details, and the other temporal motion frames (TMF) at a lower spatial resolution that retain the motion smoothness; then compress them together using any popular video coder; and finally synthesize decoded STFs and TMFs for high-fidelity video reconstruction at the same resolution as its native input. This work simply applies the bicubic resampling in decomposition and HEVC compliant codec in compression, and puts the focus on the synthesis part. For resolution-adaptive synthesis, a motion compensation network (MCN) is devised on TMFs to efficiently align and aggregate temporal motion features that will be jointly processed with corresponding STFs using a non-local texture transfer network (NL-TTN) to better augment spatial details, by which the compression and resolution resampling noises can be effectively alleviated with better rate-distortion efficiency. Such "Decomposition, Compression, Synthesis (DCS)" based scheme is codec agnostic, currently exemplifying averaged $\approx$1 dB PSNR gain or $\approx$25% BD-rate saving, against the HEVC anchor using reference software. In addition, experimental comparisons to the state-of-the-art methods and ablation studies are conducted to further report the efficiency and generalization of DCS algorithm, promising an encouraging direction for future video coding.

Figures (7)

• Figure 1: DCS-based Video Coding.${\downarrow}$ (in red) is for down-sampling , and $\uparrow$ (in red) is for up-sampling; E and D represent video encoder and decoder respectively.
• Figure 2: Motion Compensation Network (MCN). A deformable convolution is used for temporal feature alignment across neighboring frames with an auto-encoder network to generate the feature offsets; and separable temporal-spatial attentions are consecutively augmented to aggregate most proper motion features. DCN stands for the deformable convolution network; T, H/2, W/2, and C are the dimensional size of feature maps.
• Figure 3: Conditional Convolutions. Conditional weights and biases are trained according to the QP offset settings.
• Figure 4: Non-Local Texture Transfer Network (NL-TTN). (a) Semantically similarity generation: T denotes the transpose operation, U is for unfolding, and S is index selection; (b) Similarity-driven fusion.
• Figure 5: Rate-distortion curves of proposed DCS, RCAN Zhang_2018_ECCV, EDVR Wang_2019_CVPR_Workshops, bicubic-based video coding, HEVC anchor using reference model - HEVC-HM, and an additional HEVC encoder x265 - HEVC-x265.