RetimeGS: Continuous-Time Reconstruction of 4D Gaussian Splatting

Abstract

Temporal retiming, the ability to reconstruct and render dynamic scenes at arbitrary timestamps, is crucial for applications such as slow-motion playback, temporal editing, and post-production. However, most existing 4D Gaussian Splatting (4DGS) methods overfit at discrete frame indices but struggle to represent continuous-time frames, leading to ghosting artifacts when interpolating between timestamps. We identify this limitation as a form of temporal aliasing and propose RetimeGS, a simple yet effective 4DGS representation that explicitly defines the temporal behavior of the 3D Gaussian and mitigates temporal aliasing. To achieve smooth and consistent interpolation, we incorporate optical flow-guided initialization and supervision, triple-rendering supervision, and other targeted strategies. Together, these components enable ghost-free, temporally coherent rendering even under large motions. Experiments on datasets featuring fast motion, non-rigid deformation, and severe occlusions demonstrate that RetimeGS achieves superior quality and coherence over state-of-the-art methods.

Figures (10)

• Figure 1: We introduce a 4DGS representation with tailored training strategies that enables interpolating arbitrary intermediate frames, even under relatively large inter-frame motion. Our method reconstructs high-quality frames in challenging scenarios characterized by non-rigid deformations, complex textures, and visibility changes. Project page: https://william-wang2.github.io/RetimeGS/.
• Figure 2: Illustration of temporal overfitting to input frames $t_1$ and $t_2$, causing ghosting at $t_{1.5}$ in 4D primitive-based methods.
• Figure 3: Pipeline Overview. We represent a dynamic scene using a novel 4D representation that combines regularized temporal opacity with smooth spline-based spatial positioning. By leveraging tailored training strategies using RGB images and bidirectional optical flow, our method can reconstruct arbitrary intermediate frames under sparse temporal sampling and large motion.
• Figure 4: Qualitative comparison. Results on DNA-Rendering Dataset 2023dnarendering (w/o GT held-out views, left) and Stage-Capture Dataset (w/ GT held-out views, right). The red boxes show the corresponding zoomed-in views for detailed comparison.
• Figure 5: (\ref{['fig:ablation_flow']}) The texture is distorted when flow-related components are removed. (\ref{['fig:ablation_triple_render']}) Without triple rendering, the two adjacent primitive groups each capture only part of the content in the input frames they jointly cover: the previous group reconstructs the right texture in the circled region, while the next group reconstructs the left texture.