Real-Time Neural Video Compression with Unified Intra and Inter Coding

TL;DR

This work tackles real-time neural video compression by unifying intra- and inter-frame coding within a single model and introducing simultaneous two-frame compression to exploit forward and backward temporal redundancy. The proposed UI^2C framework employs a two-frame quantization scheme and hybrid-reference training to adaptively balance coding modes without manual refresh. It achieves BD-rate reductions of up to 12.1% over DCVC-RT and 35.7% over VTM while maintaining real-time encoding/decoding speeds, and it demonstrates robust performance across scene changes without bitrate spikes. Overall, the approach improves resilience to disocclusion and error propagation, offering a practical path toward deployable real-time neural video compression.

Abstract

Neural video compression (NVC) technologies have advanced rapidly in recent years, yielding state-of-the-art schemes such as DCVC-RT that offer superior compression efficiency to H.266/VVC and real-time encoding/decoding capabilities. Nonetheless, existing NVC schemes have several limitations, including inefficiency in dealing with disocclusion and new content, interframe error propagation and accumulation, among others. To eliminate these limitations, we borrow the idea from classic video coding schemes, which allow intra coding within inter-coded frames. With the intra coding tool enabled, disocclusion and new content are properly handled, and interframe error propagation is naturally intercepted without the need for manual refresh mechanisms. We present an NVC framework with unified intra and inter coding, where every frame is processed by a single model that is trained to perform intra/inter coding adaptively. Moreover, we propose a simultaneous two-frame compression design to exploit interframe redundancy not only forwardly but also backwardly. Experimental results show that our scheme outperforms DCVC-RT by an average of 12.1% BD-rate reduction, delivers more stable bitrate and quality per frame, and retains real-time encoding/decoding performances. Code and models will be released.

Figures (6)

• Figure 1: Bitrate and quality variation across frames. The test video is Kimono1 from HEVC Class B, which contains a scene change at the 141st frame. Average bitrates in bpp: DCVC-RT (no refresh) 0.0203, DCVC-RT (with refresh) 0.0182, and ours 0.0172. Average PSNR in dB: DCVC-RT (no refresh) 37.33, DCVC-RT (with refresh) 39.33, and ours 39.57. Intra-period is set to -1; DCVC-RT (with refresh) uses a refresh period of 64; our scheme needs no refresh mechanism.
• Figure 2: Framework of our neural video compression model with unified intra and inter coding. $x_0,x_1,x_t,x_{t+1}$ denote the first, second, $(t+1)$-th, $(t+2)$-th frames of the video sequence, and $\hat{x}_0,\dots$ are the corresponding reconstructed frames, respectively. $AD_I$ is the first-frame-referenced adaptor that accepts a blank signal as input, $FE$ stands for the feature extractor, and Codec denotes the encoder and decoder network. $F_{(t, t+1)}$ represents the intermediate features generated during the encoding/decoding of $x_t$ and $x_{t+1}$, while $C_{(t, t+1)}$ denotes the reference features for $x_t$ and $x_{t+1}$.
• Figure 3: Detailed architecture of our proposed UI$^2$C (unified intra and inter coding) scheme. $(E_1, E_2, E_3)$, $(D_1, D_2)$, $(R_1, R_2)$, $(FE_1, FE_1)$ represent the components of the encoder, decoder, reconstruction generator, and feature extractor, respectively (detailed configurations are provided in the supplementary material). The reference feature $F_{(t-2, t-1)}$ is used to obtain the contextual feature $C^e_{(t, t+1)}$ and $C_{(t, t+1)}$. The quantization parameters $qp_1$ and $qp_2$ are used to select the respective quantization tables. $\times$ stands for element-wise multiply.
• Figure 4: Rate-distortion curves for HEVC Class E and HEVC Class B. Color space is YUV420; all frames are coded; intra-period is -1. Results on more datasets are provided in the supplementary material.
• Figure 5: Rate-distortion curves for HEVC Class B, HEVC Class E, and UVG when only the first two frames are coded. DCVC-RT-I-intra uses the normal DCVC-RT setting (the first frame uses the intra coding model, and the second frame uses the inter coding model). DCVC-RT-P-intra only uses the DCVC-RT inter coding model, i.e., we insert a blank frame (all zeros) before the first frame, and do not count the bitrate/PSNR of the blank frame. Our method using a single model to compress the two frames simultaneously performs slightly worse than DCVC-RT-I-intra, which uses two models for intra and inter coding, respectively, but performs much better than DCVC-RT-P-intra.