GS2POSE: Marry Gaussian Splatting to 6D Object Pose Estimation

TL;DR

GS2POSE addresses 6D pose estimation for textureless objects by combining a coarse, NOCS-based pose estimator with a differentiable, GS-based refinement pipeline. It introduces 3D Gaussian Splatting to represent geometry, a ray-projection ICP for depth alignment, and an illumination-adaptive color model (GS-light) to handle lighting changes, all guided by Lie algebra-based pose updates. The method achieves state-of-the-art or near-state-of-the-art results on T-LESS, LineMod-Occlusion, and LineMod, while avoiding synthetic data dependencies and providing a training-free refinement stage. Despite its strengths, GS2POSE remains instance-level and relatively slow due to iterative optimization, suggesting avenues for generalization to unseen objects and faster convergence in future work.

Abstract

Accurate 6D pose estimation of 3D objects is a fundamental task in computer vision, and current research typically predicts the 6D pose by establishing correspondences between 2D image features and 3D model features. However, these methods often face difficulties with textureless objects and varying illumination conditions. To overcome these limitations, we propose GS2POSE, a novel approach for 6D object pose estimation. GS2POSE formulates a pose regression algorithm inspired by the principles of Bundle Adjustment (BA). By leveraging Lie algebra, we extend the capabilities of 3DGS to develop a pose-differentiable rendering pipeline, which iteratively optimizes the pose by comparing the input image to the rendered image. Additionally, GS2POSE updates color parameters within the 3DGS model, enhancing its adaptability to changes in illumination. Compared to previous models, GS2POSE demonstrates accuracy improvements of 1.4\%, 2.8\% and 2.5\% on the T-LESS, LineMod-Occlusion and LineMod datasets, respectively.

Figures (5)

• Figure 1: The pipeline of the GS2POSE. (a) 3D reconstruction. Before pose estimation, the 3D model of the object is constructed using 3DGS. (b) Coarse Pose Estimator. In this stage, the new view image and its corresponding mask (generated by CNOS model) are fed into the Pose-Unet. This network can generate the Normalized Object Coordinate Space (NOCS) image corresponding to the current viewpoint. By applying a PnP+RANSAC algorithm on the NOCS image, the coarse pose estimation transformation $T_{coarse}$ can be obtained. (c) Pose Refiner. In this stage, the point cloud generated from the RGBD image $P_{target}$ and the point cloud generated by ray projection $P_{source}$ are first processed by the ICP algorithm. ICP can perform an initial refinement of $T_{coarse}$, yielding $T_{refine}$. Subsequently, $T_{refine}$ is fed into the Camera Refinement and Object Refinement to get an accurate pose estimation result $T_{precise}$.
• Figure 2: The principle of the Camera refiner and Object refiner. Upon inputting the pose estimation transformation $T_{refine}$, the translation parameters undergo initial refinement through the Camera Refiner. Subsequently, the rotation parameters are corrected during the Object Refiner phase.
• Figure 3: Visualization of the GS2POSE process. The original image is first processed through the CNOS model to generate the corresponding mask image. Subsequently, both the original image and the mask are input into the model, which are then processed by the coarse Pose-Unet to produce the NOCS image. The coarse pose estimation transformation can be directly derived from the NOCS images. Following this, the coarse pose estimation transformation is utilized as input for the GS-refiner to achieve precise pose estimation. In the GS-refiner stage, the images presented in the top row represent real-view images, while those in the bottom row are rendered outputs generated by the GS-refiner.
• Figure 4: Visualization Results of the Pose-Unet Network: (a) represents the original images, (b) represents the ground truth of the NOCS images, (c) represents the NOCS images generated by the Pose-Unet network.
• Figure 5: The angular prediction error distributions of the Coarse Pose Estimator and the Pose Refiner.