Object-centric Reconstruction and Tracking of Dynamic Unknown Objects using 3D Gaussian Splatting

TL;DR

This work proposes a novel approach to incrementally reconstruct and track a dynamic unknown object using a unified representation-a set of 3D Gaussian blobs that describe its geometry and appearance using the differentiable 3DGS framework.

Abstract

Generalizable perception is one of the pillars of high-level autonomy in space robotics. Estimating the structure and motion of unknown objects in dynamic environments is fundamental for such autonomous systems. Traditionally, the solutions have relied on prior knowledge of target objects, multiple disparate representations, or low-fidelity outputs unsuitable for robotic operations. This work proposes a novel approach to incrementally reconstruct and track a dynamic unknown object using a unified representation -- a set of 3D Gaussian blobs that describe its geometry and appearance. The differentiable 3D Gaussian Splatting framework is adapted to a dynamic object-centric setting. The input to the pipeline is a sequential set of RGB-D images. 3D reconstruction and 6-DoF pose tracking tasks are tackled using first-order gradient-based optimization. The formulation is simple, requires no pre-training, assumes no prior knowledge of the object or its motion, and is suitable for online applications. The proposed approach is validated on a dataset of 10 unknown spacecraft of diverse geometry and texture under arbitrary relative motion. The experiments demonstrate successful 3D reconstruction and accurate 6-DoF tracking of the target object in proximity operations over a short to medium duration. The causes of tracking drift are discussed and potential solutions are outlined.

Figures (4)

• Figure 2: Incremental reconstruction and tracking of CHEOPS (left) and SOHO (right) spacecraft from a sequence of simulated images without prior training using object-level 3D Gaussian representations and splatting-based differentiable rendering.
• Figure 3: (a) Methodology: We use differentiable rendering of 3D Gaussians as the core of our incremental reconstruction and tracking pipeline. We refine the camera pose or the object Gaussians using first-order gradient-based optimization by propagating the gradients backward through the rendering process. (b) 3D Gaussian Splatting: Illustration of projecting 3D Gaussians $\mathcal{G}_{\Sigma}$ to 2D Gaussian splats $\mathcal{G}_{\Sigma"}$.
• Figure 4: (a) Dataset objects: ESA science fleet spacecraft models used to generate the dataset. (b) Test Trajectory: Visualization of the camera trajectory relative to the target spacecraft, whose body frame is denoted by the axes at $O$. The arrows show the boresight direction at each point.
• Figure 5: Results: Pose tracking errors over 1000 frames for 10 synthetic spacecraft models of diverse geometry and textures. A sample input image and the point cloud of the first RGB-D frame for each spacecraft are shown on the left.