GIFStream: 4D Gaussian-based Immersive Video with Feature Stream

TL;DR

GIFStream tackles efficient immersive video by combining a deformation-based 4D Gaussian representation with time-dependent feature streams to capture fast motion. It introduces anchors in a canonical space carrying both time-independent features and time-dependent streams, decoded into $K$ Gaussian primitives per timestamp with motion $(\mathbf{R}_t, \mathbf{T}_t)$ and attributes, rendered via splatting. The method enables end-to-end compression by partitioning parameters into two videos, applying quantization-aware training and an auto-regressive entropy model, and employs a densified-pruning optimization strategy with multiple regularized losses. Empirical results on Neur3D, Panoptic Sports, and MPEG show GIFStream achieves state-of-the-art rate–distortion with small storage and real-time decoding (over $60$ FPS on an RTX $4090$), illustrating a practical path toward high-quality 6-DoF immersive video at scale.

Abstract

Immersive video offers a 6-Dof-free viewing experience, potentially playing a key role in future video technology. Recently, 4D Gaussian Splatting has gained attention as an effective approach for immersive video due to its high rendering efficiency and quality, though maintaining quality with manageable storage remains challenging. To address this, we introduce GIFStream, a novel 4D Gaussian representation using a canonical space and a deformation field enhanced with time-dependent feature streams. These feature streams enable complex motion modeling and allow efficient compression by leveraging temporal correspondence and motion-aware pruning. Additionally, we incorporate both temporal and spatial compression networks for end-to-end compression. Experimental results show that GIFStream delivers high-quality immersive video at 30 Mbps, with real-time rendering and fast decoding on an RTX 4090. Project page: https://xdimlab.github.io/GIFStream

Figures (10)

• Figure 1: GIFStream achieves high quality and small storage size on dynamic scenes containing fast motion. We present the rendering results on a challenging scene on the left and the Rate-SSIM curve on the right.
• Figure 2: Comparison of 4D Representations. 1) Deformation-based representation stores a 3D Gaussian in a canonical space and its deformation along a long time horizon. and 2) 4D Gaussian representation which models a windowed spacetime region. We propose 3) GIFStream by adding time-dependent feature streams on top of deformation-based representation, improving its capacity while maintaining temporal alignment for efficient compression.
• Figure 3: Method. (I. Representation) We propose enhancing deformation-based dynamic Gaussian representation using time-dependent feature streams. We attach time-dependent feature $\mathbf{f}_t$ and time-independent feature $\mathbf{f}$ to a set of anchor points. These features are aggregated to decode deformation motion $\mathbf{R}_t, \mathbf{T}_t$ and Gaussian attribute $\alpha_t, \mathbf{s}_t, \mathbf{r}_t, \mathbf{c}_t$ at a specific timestamp $t$ through MLPs. Finally, we render the target view through splatting. (II. Compression) For compression, we reorganize both time-dependent and time-independent parameters into two videos. The feature streams are first pruned and then compressed in an auto-regressive manner, effectively leveraging the temporal correspondence information. During training, we jointly optimize the rendering loss $\mathcal{L}_{photo}$ and add an entropy constraint $\mathcal{L}_{entropy}$.
• Figure 4: Motion Illustration. We predict the rotation and translations on the local coordinate system of the anchor.
• Figure 5: RD Curve Comparison on MPEG dataset. We visualize the RD Curve results in the GOP 65 setting.