GSFeatLoc: Visual Localization Using Feature Correspondence on 3D Gaussian Splatting

TL;DR

This work tackles real-time visual localization with respect to a precomputed 3D Gaussian Splatting (3DGS) scene by replacing slow photometric-loss refinement with a fast, feature-based pipeline. It renders a single RGBD image at an initial pose, establishes 2D-2D correspondences to the query, lifts these to 2D-3D using the rendered depth, and solves a PnP problem to estimate the final pose, enabling runtime as low as $0.1$ s per image. The method tolerates large initial pose errors up to $55^{\circ}$ in rotation and $1.1$ units in translation and achieves final pose errors below $5^{\circ}$ and $0.05$ units on most images across Synthetic NeRF, Mip-NeRF360, and Tanks and Temples datasets. Across three datasets and 38 scenes, the approach outperforms photometric-loss baselines in both inference time and accuracy, highlighting its potential for real-time robotics and SLAM applications where initial pose estimates are rough.

Abstract

In this paper, we present a method for localizing a query image with respect to a precomputed 3D Gaussian Splatting (3DGS) scene representation. First, the method uses 3DGS to render a synthetic RGBD image at some initial pose estimate. Second, it establishes 2D-2D correspondences between the query image and this synthetic image. Third, it uses the depth map to lift the 2D-2D correspondences to 2D-3D correspondences and solves a perspective-n-point (PnP) problem to produce a final pose estimate. Results from evaluation across three existing datasets with 38 scenes and over 2,700 test images show that our method significantly reduces both inference time (by over two orders of magnitude, from more than 10 seconds to as fast as 0.1 seconds) and estimation error compared to baseline methods that use photometric loss minimization. Results also show that our method tolerates large errors in the initial pose estimate of up to 55° in rotation and 1.1 units in translation (normalized by scene scale), achieving final pose errors of less than 5° in rotation and 0.05 units in translation on 90% of images from the Synthetic NeRF and Mip-NeRF360 datasets and on 42% of images from the more challenging Tanks and Temples dataset.

Figures (5)

• Figure 1: Our method estimates the pose of a query image on a 3D Gaussian Splatting (3DGS) scene by establishing feature correspondences between the query image and an image rendered at a rough initial pose (a). The matched points are then lifted to 3D using the depth map rendered from 3DGS (b), and the final pose is estimated by solving a PnP problem. The image rendered at the estimated pose (c) is also shown.
• Figure 2: Overview of the proposed pipeline for visual localization using 3DGS as the scene representation, given a query image $I_q$, an initial pose estimate $\bm{\mathrm{T}}^{0} \in \mathrm{SE}(3)$, and the 3DGS scene representation.
• Figure 3: Example pairs of failure and success cases from several scenes in the Tanks and Temples dataset, where success is defined as $\mathrm{RE} < 5^{\circ}$ and $\mathrm{TE} < 0.05$. Each triplet shows, from left to right, the query image, the image rendered at the initial pose provided by 6DGS, and the image rendered at the pose estimated by our method.
• Figure 4: Percentage of pose estimates (out of 200 test images) with rotation error $<5^{\circ}$ and normalized translation error $<0.05$ as a function of difference between the initial pose estimate and ground-truth in yaw on the Lego scene (unit: %). $\Delta \theta = \rm{33.84}^{\circ}$ is determined as the minimum yaw error required for the object to move out of view.
• Figure 5: Percentage of pose estimates (out of 200 test images) with rotation error $<5^{\circ}$ and normalized translation error $<0.05$ as a function of difference between the initial pose and ground-truth in x-position on the Lego scene (unit: %). $\Delta p = \rm{2.44}~\rm{m}$ is determined as the minimum translation error required for the object to move out of view.