NuRF: Nudging the Particle Filter in Radiance Fields for Robot Visual Localization

TL;DR

NuRF tackles monocular visual localization by combining radiance-field maps with a Nudged Particle Filter to achieve continuous 6-DoF localization on indoor environments. It introduces Radiance Field anchors with VPR, a nudging mechanism to steer particles toward high-likelihood regions, and an adaptive rendering scheme that switches between global localization and pose tracking. Empirical results on a blimp platform show NuRF converges faster (up to several-fold) and attains sub-meter accuracy, outperforming Loc-NeRF and pixel-based PF baselines, with ablations confirming the value of nudging. The work advances practical monocular localization on radiance-field maps, while noting limitations in texture-poor scenes and rendering speed, and suggesting directions toward faster map representations and integrated planning.”

Abstract

Can we localize a robot on a map only using monocular vision? This study presents NuRF, an adaptive and nudged particle filter framework in radiance fields for 6-DoF robot visual localization. NuRF leverages recent advancements in radiance fields and visual place recognition. Conventional visual place recognition meets the challenges of data sparsity and artifact-induced inaccuracies. By utilizing radiance field-generated novel views, NuRF enhances visual localization performance and combines coarse global localization with the fine-grained pose tracking of a particle filter, ensuring continuous and precise localization. Experimentally, our method converges 7 times faster than existing Monte Carlo-based methods and achieves localization accuracy within 1 meter, offering an efficient and resilient solution for indoor visual localization.

Figures (11)

• Figure 1: NuRF achieves monocular localization on images generated from radiance fields. The image within the blue box is observed by our blimp robot, while the image in the green box is a reference image rendered from radiance fields..
• Figure 2: The pipeline of our designed NuRF framework. We first use radiance fields to generate images on anchor poses, store these images in a database, and vectorize them for retrieval. Then, a particle filter is built for robot localization in radiance fields. The nudging step uses retrieved results (from VPR) to guide the particles toward more confident states. The measurement model updates particle weights based on images that are observed and rendered by particles. The motion model adjusts particles according to robot motion. Our adaptive workflow enables switching between global localization and pose tracking to address the robot kidnapping problem.
• Figure 3: We show the original image and the rendered image in pixel space and feature space correspondingly. In the comparison column, the difference between the original image and the rendered image is highlighted in blue. We assess the similarity between a query image and images rendered in the same orientation but shifted along the X-axis using pixel-wise similarity and feature-wise similarity.
• Figure 4: We display 504 anchor poses in 2D sub-manifold $S$, along with images rendered at these specific anchors for visualization.
• Figure 5: Pipeline of nudging particles using the feature information from VPR. Upon the observation of a new image, the ViT encoder transforms it into a feature map. This map is then vectorized and subjected to cosine distance calculations with the vectors stored in the VPR database. The camera pose corresponding to the $k$ vectors with the shortest cosine distance are retrieved and add into the particle set.