Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation

TL;DR

Split4D introduces a tracker-free decomposed 4D scene representation built on Freetime FeatureGS, where Gaussian primitives with linear motion carry learnable features that are trained via a temporally extended contrastive loss. A streaming sampling strategy propagates features over time, enabling robust 4D segmentation from per-image segmentation maps without relying on video trackers. Regularizations from 3D tracking and DINOv2 alignment further stabilize learning, while an inference-time filtering strategy refines instance boundaries. Across Neural3DV, Multi-Human, and SelfCap, Split4D achieves state-of-the-art segmentation and temporal coherence, with demonstrated applications in dynamic scene editing and monocular street-scene decomposition. This approach reduces reliance on segmentation consistency while enabling accurate, editable decomposed 4D scenes in challenging dynamic contexts.

Abstract

This paper addresses the problem of decomposed 4D scene reconstruction from multi-view videos. Recent methods achieve this by lifting video segmentation results to a 4D representation through differentiable rendering techniques. Therefore, they heavily rely on the quality of video segmentation maps, which are often unstable, leading to unreliable reconstruction results. To overcome this challenge, our key idea is to represent the decomposed 4D scene with the Freetime FeatureGS and design a streaming feature learning strategy to accurately recover it from per-image segmentation maps, eliminating the need for video segmentation. Freetime FeatureGS models the dynamic scene as a set of Gaussian primitives with learnable features and linear motion ability, allowing them to move to neighboring regions over time. We apply a contrastive loss to Freetime FeatureGS, forcing primitive features to be close or far apart based on whether their projections belong to the same instance in the 2D segmentation map. As our Gaussian primitives can move across time, it naturally extends the feature learning to the temporal dimension, achieving 4D segmentation. Furthermore, we sample observations for training in a temporally ordered manner, enabling the streaming propagation of features over time and effectively avoiding local minima during the optimization process. Experimental results on several datasets show that the reconstruction quality of our method outperforms recent methods by a large margin.

Figures (12)

• Figure 1: Overview of Our Pipeline. (a) The decomposed 4D representation is constructed based on spatio-temporally free Gaussian primitives, each associated with a learnable feature vector. (b) During training, a streaming sampling strategy selects temporally ordered frames, ensuring that features propagate smoothly along their motion trajectories across time. (c) Exploiting Freetime FeatureGS's local temporal continuity, we impose a contrastive loss on frame-wise inconsistent 2D segmentation labels, enabling 4D feature learning.
• Figure 2: Analysis of Motion Modeling and Streaming Learning. We compare our full model with variants that remove motion modeling and streaming learning strategy. The results demonstrate that both components are crucial for handling complex motions and maintaining temporal consistency in feature learning. The full model achieves the best performance, while removing either component leads to significant performance degradation.
• Figure 3: Qualitative Comparison Results. We visually compare our method with several baselines, including OmniSeg3D omniseg3d, SA4D sa4d with post-processing, and SADG sadg, as well as with Ground Truth. The results are organized by dataset, with each dataset's results spanning two rows: Neural3DV (top two rows), Multi-Human (middle two rows), and SelfCap (bottom two rows). Our method consistently achieves the best segmentation performance across all datasets, especially on Multi-Human and SelfCap, where scenes involve complex motions.
• Figure 4: Comparison of Boundary Segmentation in Human-Object Interaction. The circled regions highlight areas of hand-object contact, where our method produces significantly more accurate boundaries.
• Figure 5: Qualitative Results of Ablation Studies. We present a qualitative comparison of segmentation results for different ablation settings on the Juggle sequence of the Multi-Human dataset.