OnlineSplatter: Pose-Free Online 3D Reconstruction for Free-Moving Objects

TL;DR

OnlineSplatter is introduced, a novel online feed-forward framework generating high-quality, object-centric 3D Gaussians directly from RGB frames without requiring camera pose, depth priors, or bundle optimization.

Abstract

Free-moving object reconstruction from monocular video remains challenging, particularly without reliable pose or depth cues and under arbitrary object motion. We introduce OnlineSplatter, a novel online feed-forward framework generating high-quality, object-centric 3D Gaussians directly from RGB frames without requiring camera pose, depth priors, or bundle optimization. Our approach anchors reconstruction using the first frame and progressively refines the object representation through a dense Gaussian primitive field, maintaining constant computational cost regardless of video sequence length. Our core contribution is a dual-key memory module combining latent appearance-geometry keys with explicit directional keys, robustly fusing current frame features with temporally aggregated object states. This design enables effective handling of free-moving objects via spatial-guided memory readout and an efficient sparsification mechanism, ensuring comprehensive yet compact object coverage. Evaluations on real-world datasets demonstrate that OnlineSplatter significantly outperforms state-of-the-art pose-free reconstruction baselines, consistently improving with more observations while maintaining constant memory and runtime.

Figures (6)

• Figure 1: Outline of proposed OnlineSplatter. From the incoming stream of pose-free RGB frames, OnlineSplatter "splats" the observations into a canonical cloud of 3D Gaussians anchored to the first frame. Every new frame triggers a single, $\mathbf{O(1)}$memory-and-time update that immediately improves the reconstruction of the object. The system copes seamlessly with freely moving objects and requires neither pre-computed poses nor depth maps. The result is a continually improving 3D representation suitable for real-time applications.
• Figure 2: Overview of OnlineSplatter Pipeline. The input to our framework consists of a stream of RGB images $\{V_t\}_{t=0}^N$, where object masks $\{M_t\}_{t=0}^N$ are generated and applied to remove background on-the-fly using an off-the-shelf online video segmentation (OVS) module running alongside our framework. At each timestep $t$, OnlineSplatter processes the input frame $V_t$ by first patchifying it into patch tokens. These tokens are then fed into a transformer-based architecture, which directly reasons and outputs pixel-aligned 3D Gaussian representations in a canonical space. Central to our method is object memory, an implicit module based on cross-attention, which is queried and updated at every timestep. This memory enables the incremental reconstruction of the object, consistently refining the object representation ($\mathbf{G}_{obj,t}^{4N}$) as new observations arrive in a fully feed-forward manner.
• Figure 3: Qualitative results of different baselines and our method on the GSO (left) and HO3D (right) datasets. We visualize the results at inference timestep $t=4$ and $t=16$, which corresponds to the early-stage and late-stage settings, respectively. Our reconstructed outputs show significantly better visual quality and geometric accuracy as more observations become available.
• Figure 4: Visual comparison of mesh results between different methods. Methods marked with an asterisk (*) indicate that additional pre- or post-processing steps were applied to generate the visual results. More details in the appendix.
• Figure 5: Impact of Training Data Quantity and Quality.