GTR: Gaussian Splatting Tracking and Reconstruction of Unknown Objects Based on Appearance and Geometric Complexity

TL;DR

This work addresses 6-DoF object tracking and high-quality 3D reconstruction from monocular RGB-D video, proposing an adaptive pipeline that combines 3D Gaussian Splatting with appearance/geometry-aware tracking and strategic keyframe selection. It introduces a new GTR3D benchmark with full-view annotations to evaluate tracking and reconstruction on challenging object classes, including axis-symmetric and highly detailed geometries. The method leverages a multi-stage refinement strategy, including ICP, 3DGS render-and-compare, and pose-graph optimization, yielding accurate poses and detailed textured meshes, validated on synthetic and real data. The approach advances single-sensor object-centric reconstruction in open-world settings, facilitating applications in AR, robotics, and sim-to-real transfer.

Abstract

We present a novel method for 6-DoF object tracking and high-quality 3D reconstruction from monocular RGBD video. Existing methods, while achieving impressive results, often struggle with complex objects, particularly those exhibiting symmetry, intricate geometry or complex appearance. To bridge these gaps, we introduce an adaptive method that combines 3D Gaussian Splatting, hybrid geometry/appearance tracking, and key frame selection to achieve robust tracking and accurate reconstructions across a diverse range of objects. Additionally, we present a benchmark covering these challenging object classes, providing high-quality annotations for evaluating both tracking and reconstruction performance. Our approach demonstrates strong capabilities in recovering high-fidelity object meshes, setting a new standard for single-sensor 3D reconstruction in open-world environments.

Figures (27)

• Figure 1: We present GTR, an adaptive method for 6-DoF object tracking and 3D reconstruction from monocular RGBD video. Although this shows the camera's trajectory based on the object coordinate, the fixed camera is used for the tracking of moving objects.
• Figure 2: Pipeline Flow: Our method processes sequential RGB-D frames and estimates coarse relative poses using the (1) Keypoint Detection & Tracking module. If the poses meet the visibility criteria, the frames are added to the keyframe pool (2). To refine the keyframe poses, the (3) Pose Refinement module—incorporating ICP, 3DGS-based render-and-compare, and pose graph optimization techniques—is applied. Finally, shapes are extracted via TSDF fusion using the recovered poses.
• Figure 3: Appearance and Geometric Complexity: The detected keypoints from SIFT are visualized in the left figure. The appearance complexity can vary depending on the viewpoint, even for the same object. Lower complexity makes keypoint tracking more difficult. We also visualize the geometric complexity at each pixel using a color map based on our proposed method. Higher complexity makes the registration problem easier.
• Figure 4: Coarse Pose Solver: We use TEASER++ as part of our keypoint detection and tracking module to obtain an initial pose estimate for the new frame relative to the last keyframe using coarse correspondences.
• Figure 5: Keyframe Selection: We use the visibility rate across the multiple images based on 2D keypoint tracking as a criteria for keyframe selection. The number of visible key points that is sampled on the left frame decreases when the target object moves.