Neural Video Compression using 2D Gaussian Splatting

TL;DR

This work tackles real-time neural video compression by introducing an explicit, ROI-focused 2D Gaussian Splatting representation. It integrates content-aware ROI initialization, I-/P-frame structure with metadata-driven selective optimization, and quantization/entropy coding to balance speed and quality. Key findings include an 8× encoding-time speedup over prior Gaussian Splatting image codecs, substantial bitrate savings over the previous Gaussian approach, and configurable rate-distortion performance that bridges toward INR-based codecs while maintaining frame-by-frame transmission. The approach advances practical neural video coding for ROI-centric applications and motivates future work toward real-time, hardware-friendly solutions.

Abstract

The computer vision and image processing research community has been involved in standardizing video data communications for the past many decades, leading to standards such as AVC, HEVC, VVC, AV1, AV2, etc. However, recent groundbreaking works have focused on employing deep learning-based techniques to replace the traditional video codec pipeline to a greater affect. Neural video codecs (NVC) create an end-to-end ML-based solution that does not rely on any handcrafted features (motion or edge-based) and have the ability to learn content-aware compression strategies, offering better adaptability and higher compression efficiency than traditional methods. This holds a great potential not only for hardware design, but also for various video streaming platforms and applications, especially video conferencing applications such as MS-Teams or Zoom that have found extensive usage in classrooms and workplaces. However, their high computational demands currently limit their use in real-time applications like video conferencing. To address this, we propose a region-of-interest (ROI) based neural video compression model that leverages 2D Gaussian Splatting. Unlike traditional codecs, 2D Gaussian Splatting is capable of real-time decoding and can be optimized using fewer data points, requiring only thousands of Gaussians for decent quality outputs as opposed to millions in 3D scenes. In this work, we designed a video pipeline that speeds up the encoding time of the previous Gaussian splatting-based image codec by 88% by using a content-aware initialization strategy paired with a novel Gaussian inter-frame redundancy-reduction mechanism, enabling Gaussian splatting to be used for a video-codec solution, the first of its kind solution in this neural video codec space.

Figures (8)

• Figure 1: Pipeline overview of our I-frame representation: region of interest extracted from input frame, passed through our novel Gaussian initializer to get content-aware initialization. This initial Gaussian state goes through optimization steps to get final render, followed by compression. Figure also shows previous work's random initialization
• Figure 2: Our novel Gaussian attribute initialization strategy: (a) K-Means clustered superpixel segments, (b) Gaussian initialization computed from superpixel segments.
• Figure 3: P-Frame representation workflow: residual pixels between two frames used to compute Gaussians influencing those pixels, which are then selectively optimized followed by compression
• Figure 4: Encoding and decoding latency comparison of different codecs on VCD dataset
• Figure 5: Bits-per-pixel comparison of our codec with previous GS codec