Gaussian Time Machine: A Real-Time Rendering Methodology for Time-Variant Appearances

TL;DR

Gaussian Time Machine (GTM) tackles real-time novel view synthesis for scenes with long-term appearance changes by integrating discrete time embeddings into a 3D Gaussian Splatting framework. It predicts time-variant properties of Gaussian primitives through a lightweight encoder, while a decomposed color model separates static geometry from dynamic lighting, and an adjustable opacity mechanism handles changing visibility. GTM demonstrates state-of-the-art rendering fidelity on three real-world datasets, runs at about 80 FPS, and uses significantly less storage than NeRF-based approaches, all while enabling smooth appearance interpolation. The approach combines the efficiency of 3DGS with robust handling of discrete time appearance variations, offering practical impact for interactive visualization, street-view time machines, and digital twins, though it notes future work on enforcing physical consistency during varying appearances.

Abstract

Recent advancements in neural rendering techniques have significantly enhanced the fidelity of 3D reconstruction. Notably, the emergence of 3D Gaussian Splatting (3DGS) has marked a significant milestone by adopting a discrete scene representation, facilitating efficient training and real-time rendering. Several studies have successfully extended the real-time rendering capability of 3DGS to dynamic scenes. However, a challenge arises when training images are captured under vastly differing weather and lighting conditions. This scenario poses a challenge for 3DGS and its variants in achieving accurate reconstructions. Although NeRF-based methods (NeRF-W, CLNeRF) have shown promise in handling such challenging conditions, their computational demands hinder real-time rendering capabilities. In this paper, we present Gaussian Time Machine (GTM) which models the time-dependent attributes of Gaussian primitives with discrete time embedding vectors decoded by a lightweight Multi-Layer-Perceptron(MLP). By adjusting the opacity of Gaussian primitives, we can reconstruct visibility changes of objects. We further propose a decomposed color model for improved geometric consistency. GTM achieved state-of-the-art rendering fidelity on 3 datasets and is 100 times faster than NeRF-based counterparts in rendering. Moreover, GTM successfully disentangles the appearance changes and renders smooth appearance interpolation.

Figures (6)

• Figure 1: Gaussian Time Machine (GTM) addresses the challenge of reconstructing time-variant 3D scenes with complicated appearance changes using training images taken at discrete moments. We apply a lightweight neural time encoder, empowering 3DGS to disentangle scene variations. GTM has combined strengths of high quality, real-time speed, and flexibility in handling both lighting changes and transformations of objects, only with a single encoder and 3D Gaussian point cloud.
• Figure 2: Overview of Gaussian Time Machine(GTM). A set of lightweight neural networks take discrete time embedding vectors as input, predicting the time-variant properties of neural Gaussian primitives. Then neural Gaussians are blended to render images. The network parameters and embedding vectors are jointly optimized by minimizing the loss function between the predicted images and ground-truth images, along with a regularization term.
• Figure 3: Visual comparison of rendering quality between GTM and existing methods on WATCai_2023_ICCV dataset (first 3 rows) and NeRF-OSRosr dataset (the rest). NeRF-based approaches with time embeddings can disentangle appearance changes but limited representation capability often leads to floater artifacts. Continuous-time 3DGS methods have stronger representation capability, but due to the limitation of their time encoding scheme, they tend to overfit training images, or cannot reconstruct complex variations, leading to blurry artifacts. GTM combines the strengths of the two approaches and can produce high-fidelity renderings, especially on details and distant parts of the scene.
• Figure 4: Visual comparison between 4DGS and GTM on appearance control. We select two scenes from WAT. For GTM, we linearly interpolate the embedding vector. For 4DGS, we train on discrete time stamps and assign a continuous sequence of time steps in between. Interpolation on GTM produces consistent geometry over time and the trajectories of most Gaussian primitives are ordered, so there are less artifacts. These results demonstrate that GTM's time encoding scheme can disentangle the variance of color from the invariant part of the scene.
• Figure 5: Cases where GTM reconstruction results are inconsistent with physical movement. Dynamic opacity can accurately control the visibility of the desired part of the scene, but the interpolation results are gradual fading-in and fading-out of specific parts, which is inconsistent with the true movement trajectory.