NVC-1B: A Large Neural Video Coding Model

TL;DR

This work investigates scaling neural video coding models by expanding a small baseline into a billion-parameter architecture (NVC-1B). By systematically scaling individual modules (motion encoding/decoding, entropy models, contextual coding, and temporal context mining) and exploring CNN, mixed CNN-Transformer, and Transformer designs, the authors identify where gains come from and how stability constrains growth. The resulting NVC-1B achieves significant rate–distortion improvements over the small baseline and surpasses state-of-the-art neural codecs on RGB and YUV420 data, demonstrating the viability and impact of large neural video codecs. The findings suggest that large, well-structured CNN-based models can push video compression efficiency toward or beyond traditional standards, highlighting a path for future scalable video coding research.

Abstract

The emerging large models have achieved notable progress in the fields of natural language processing and computer vision. However, large models for neural video coding are still unexplored. In this paper, we try to explore how to build a large neural video coding model. Based on a small baseline model, we gradually scale up the model sizes of its different coding parts, including the motion encoder-decoder, motion entropy model, contextual encoder-decoder, contextual entropy model, and temporal context mining module, and analyze the influence of model sizes on video compression performance. Then, we explore to use different architectures, including CNN, mixed CNN-Transformer, and Transformer architectures, to implement the neural video coding model and analyze the influence of model architectures on video compression performance. Based on our exploration results, we design the first neural video coding model with more than 1 billion parameters -- NVC-1B. Experimental results show that our proposed large model achieves a significant video compression performance improvement over the small baseline model, and represents the state-of-the-art compression efficiency. We anticipate large models may bring up the video coding technologies to the next level.

Figures (12)

• Figure 1: Overview of our proposed large neural video coding model--NVC-1B. We explore to scale up the model size motion encoder-decoder, motion entropy model, contextual encoder-decoder, contextual entropy model, and temporal context mining module. Based on our exploration results, we allocate most model parameters to the modules marked with red stars.
• Figure 2: (a) Effectiveness of scaling up the model size of motion encoder-decoder. Based on our small baseline model sheng2024spatial with 21M parameters, three models ($M_1$, $M_2$, $M_3$) with 52M, 71M, and 96M parameters respectively are built. (b) Effectiveness of scaling up the model size of motion entropy model. Based on $M_1$, $M_2$, $M_3$, three models ($M_4$, $M_5$, $M_6$) with 128M, 177M, and 226M parameters respectively are built. When calculating the BD-rate, the anchor is our small baseline model.
• Figure 3: (a) Effectiveness of scaling up the model size of contextual encoder-decoder. Based on our small baseline model with 21M parameters, three models ($M_7$, $M_8$, $M_9$) with 50M, 77M, and 90M parameters respectively are built. (b) Effectiveness of scaling up the model size of contextual entropy model. Based on $M_7$, $M_8$, $M_9$, three models ($M_{10}$, $M_{11}$, $M_{12}$) with 92M, 199M, and 212M parameters respectively are built. When calculating the BD-rate, the anchor is our small baseline model.
• Figure 4: (a) Effectiveness of scaling up the model size of temporal context mining module. Based on our small baseline model with 21M parameters, three models ($M_{13}$, $M_{14}$, $M_{15}$) with 26M, 47M, and 75M parameters respectively are built. (b) To explore whether the gain of scaling up the temporal context mining module can be superimposed with the gain of scaling up the contextual encoder-decoder and contextual entropy model, based on $M_{10}$, $M_{11}$, $M_{12}$, three models ($M_{16}$, $M_{17}$, $M_{18}$) with 144M, 252M, and 265M parameters respectively are built. When calculating the BD-rate, the anchor is our small baseline model.
• Figure 5: Effectiveness of the mixed CNN-Transformer architecture. Based on our small baseline model with 21M parameters, we gradually insert SwinTransformer layers liu2021Swin into the contextual encoder-decoder, contextual entropy model, and temporal context mining module with CNN architectures and build three models ($M_{19}$, $M_{20}$, $M_{21}$) with 42M, 240M, and 267M parameters respectively. For comparison, we also list the compression results of their CNN-architecture counterparts with similar model sizes. When calculating the BD-rate, the anchor is our small baseline model.