Time of the Flight of the Gaussians: Optimizing Depth Indirectly in Dynamic Radiance Fields

TL;DR

This work tackles depth reconstruction in dynamic scenes from monocular continuous-wave ToF data by optimizing a dynamic Gaussian splatting representation directly against raw C-ToF measurements. It identifies that depth is only indirectly constrained in this setting and introduces two heuristics—occupancy bias and low-reflectivity bias—to stabilize optimization when using fast Gaussian splatting. The proposed method achieves approximately a 100× speedup in both optimization and rendering while delivering comparable or superior accuracy to neural radiance-field baselines, demonstrated on synthetic and real TöRF/F-TöRF datasets, including fast motion like swinging bats. The approach enables practical single-camera 4D dynamic scene reconstruction under constrained sensing, with potential for broader applicability to ToF-based dynamic scene understanding.

Abstract

We present a method to reconstruct dynamic scenes from monocular continuous-wave time-of-flight (C-ToF) cameras using raw sensor samples that achieves similar or better accuracy than neural volumetric approaches and is 100x faster. Quickly achieving high-fidelity dynamic 3D reconstruction from a single viewpoint is a significant challenge in computer vision. In C-ToF radiance field reconstruction, the property of interest-depth-is not directly measured, causing an additional challenge. This problem has a large and underappreciated impact upon the optimization when using a fast primitive-based scene representation like 3D Gaussian splatting, which is commonly used with multi-view data to produce satisfactory results and is brittle in its optimization otherwise. We incorporate two heuristics into the optimization to improve the accuracy of scene geometry represented by Gaussians. Experimental results show that our approach produces accurate reconstructions under constrained C-ToF sensing conditions, including for fast motions like swinging baseball bats. https://visual.cs.brown.edu/gftorf

Figures (11)

• Figure 1: Measuring the depth of a fast-moving object is challenging for C-ToF cameras. By modeling raw C-ToF frames, our method can reconstruct the geometry and motion of a fast swinging baseball bat. It is 100$\times$ faster to optimize and render and achieves similar or better accuracy than prior neural volumetric approaches.
• Figure 2: Fitting C-ToF images $\neq$ fitting depth.Top left: Camera-derived depth from C-ToF. Top right: Rendering a GS scene reconstruction into C-ToF raw image samples, then deriving depth. As this is similar to the camera-derived depth to the left, the reconstruction objective was met. Bottom left: Rendered mean scene depth from Gaussians, which is highly inaccurate. Bottom right: Depth distortion error huang20242d, which measures the Gaussian sparsity along each ray. Gaussians are not well localized.
• Figure 3: Pipeline.Left: We capture input raw quads (or phasors, not shown) from a continuous-wave time-of-flight camera with optional color camera. Right to Left: From randomly initialized Gaussians, the warm-up stage estimates canonical scene geometry for a static scene. Then, given time $t$, the MLP predicts offsets ($\delta \mathbf{x}_k$) that reposition the canonical Gaussians. Then, we render the C-ToF and color images and compute losses.
• Figure 4: Our approach is competitive or better in terms of accuracy against the state of the art on synthetic scenes, while being two orders of magnitude faster. We use the synthetic dataset from F-TöRF to demonstrate this. All images show rendered volumetric depth $d$. State-of-the-art NeRF model F-TöRF produces similar results to ours while being significantly slower. DeformableGS with C-ToF depth (\ref{['eq:tof_depth']}) fails to reconstruct dynamic objects well. Baseline 2D Flowed model is fast but cannot handle axial motion as well as our model, producing artifacts. Inset on ground truth: corresponding RGB image. See our supplement for larger images.
• Figure 5: F-TöRF real scenes; rendered scene depth $d$. As it models temporal dynamics and constrains geometry appropriately during training, our model produces comparable quality reconstructions with F-TöRF while being consistently better than other baselines. Our method tends to reconstruct static geometry better (floor in JumpingJacks). Fan scene presents a significant challenge as the motion is nonlinear---our canonical field struggles to track this geometry over time. F-TöRF can approximate this scene better due to the lack of such a global constraint. Depth wrapping effects in the background occur in some scenes; these represent an ambiguity that is out of our scope for this work.