GSVC: Efficient Video Representation and Compression Through 2D Gaussian Splatting

TL;DR

GSVC, an approach to learning a set of 2D Gaussian splats that can effectively represent and compress video frames, and achieves good rate-distortion trade-offs, comparable to state-of-the-art video codecs such as AV1 and VVC.

Abstract

3D Gaussian splats have emerged as a revolutionary, effective, learned representation for static 3D scenes. In this work, we explore using 2D Gaussian splats as a new primitive for representing videos. We propose GSVC, an approach to learning a set of 2D Gaussian splats that can effectively represent and compress video frames. GSVC incorporates the following techniques: (i) To exploit temporal redundancy among adjacent frames, which can speed up training and improve the compression efficiency, we predict the Gaussian splats of a frame based on its previous frame; (ii) To control the trade-offs between file size and quality, we remove Gaussian splats with low contribution to the video quality; (iii) To capture dynamics in videos, we randomly add Gaussian splats to fit content with large motion or newly-appeared objects; (iv) To handle significant changes in the scene, we detect key frames based on loss differences during the learning process. Experiment results show that GSVC achieves good rate-distortion trade-offs, comparable to state-of-the-art video codecs such as AV1 and VVC, and a rendering speed of 1500 fps for a 1920x1080 video.

Figures (7)

• Figure 1: Using Gaussian splats to model an image. Image is taken from UVG Jockey video. The sub-figures show that as the number of Gaussian splats $N$ increase, the learned splats increasing approximates the image's content
• Figure 2: Image taken from UVG Jockey video. Intermediate training results after $t$ iterations for 10,000 Gaussian splats, illustrating how Gaussian splats parameters are optimized to fit the content of a frame.
• Figure 3: Distribution of Gaussian Splat Centers ($45K$) on $f_1$ in Beauty, HoneyBee and Jockey.
• Figure 4: Distribution of Gaussian Splats ($45k$) on $f_5$ in Beauty.
• Figure 5: $\Delta\boldsymbol{\ell}$ and PSNR over frames on the concatenated videos.