Augmented Deep Contexts for Spatially Embedded Video Coding

TL;DR

This work introduces SEVC, a spatially embedded neural video codec that augments a base temporal codec with low‑resolution spatial references. By combining Motion and Feature Co‑Augmentation to refine motion vectors and features and a spatial‑guided latent prior that fuses multiple temporal latents via Transformers, SEVC produces rich hybrid contexts and improved entropy modeling. A joint spatial‑temporal optimization learns quality‑adaptive bit allocation for spatial references, yielding substantial bitrate savings over prior state‑of‑the‑art while providing an additional low‑resolution bitstream. The approach effectively mitigates challenges from large motions and emerging objects, offering practical gains for high‑fidelity video coding with scalable decoding options.

Abstract

Most Neural Video Codecs (NVCs) only employ temporal references to generate temporal-only contexts and latent prior. These temporal-only NVCs fail to handle large motions or emerging objects due to limited contexts and misaligned latent prior. To relieve the limitations, we propose a Spatially Embedded Video Codec (SEVC), in which the low-resolution video is compressed for spatial references. Firstly, our SEVC leverages both spatial and temporal references to generate augmented motion vectors and hybrid spatial-temporal contexts. Secondly, to address the misalignment issue in latent prior and enrich the prior information, we introduce a spatial-guided latent prior augmented by multiple temporal latent representations. At last, we design a joint spatial-temporal optimization to learn quality-adaptive bit allocation for spatial references, further boosting rate-distortion performance. Experimental results show that our SEVC effectively alleviates the limitations in handling large motions or emerging objects, and also reduces 11.9% more bitrate than the previous state-of-the-art NVC while providing an additional low-resolution bitstream. Our code and model are available at https://github.com/EsakaK/SEVC.

Figures (20)

• Figure 1: Compared with previous Neural Video Codecs (NVCs), our proposed codec embeds a base temporal-only codec to compress the low-resolution motion $v_t^b$ and frame $x_t^b$ and provide additional spatial references. Spatial and temporal references then augment deep contexts for frame coding.
• Figure 2: Comparison of our Spatially Embedded Video Codec (SEVC) and previous NVCs. (a) compresses the residual between the prediction frame $\bar{x}_t$ and the input frame $x_t$. (b) extracts deep contexts $C_t$ from the previously reconstructed frame to serve as the coding condition. (c) further improves the utilization of temporal references in both motion coding and frame coding. (d) conducts a base coding on the low-resolution input frame $x_t^b$ and utilizes both spatial and temporal references for frame coding.
• Figure 3: Overview of our SEVC. Our SEVC embeds a base temporal-only codec for obtaining spatial references: base MVs $\hat{v}_t^b$, spatial feature $\hat{F}_t^b$, and spatial latent representation $\hat{y}_t^b$. Both the spatial and temporal references are utilized for the coding of the input frame $x_t$.
• Figure 4: Diagrams of the Motion and Feature Co-Augmentation module. "UP" represents upsampling using subpixel convolution shi2016real with a factor of 2. "ODA" represents Offset Diversity Alignment proposed in dc. Ⓒ indicates channel dimension concatenation. $\bigoplus$ indicates element-wise addition.
• Figure 5: Visualization of base MVs and the spatial feature within the multi-stage augmentation in the middle scale.