Self-supervised Learning of Neural Implicit Feature Fields for Camera Pose Refinement

TL;DR

The paper addresses visual localization by eliminating reliance on fixed encoders or explicit SfM maps, instead proposing a self-supervised framework that jointly learns a neural implicit geometry field and a dense 3D feature field together with a 2D image encoder in a shared embedding space. It leverages volumetric rendering and contrastive learning to align rendered volumetric features with image features, enabling feature-metric pose refinement from an initial estimate. Key contributions include end-to-end training of encoder and feature field without supervision, coupling the feature field to intermediate geometry through a prototypical auxiliary loss and multiview constraints, and achieving state-of-the-art performance among implicit representations on Cambridge Landmarks with competitive results on 7-Scenes. The approach provides a scalable, geometry-grounded pathway to robust localization that can generalize beyond fixed 3D maps and traditional feature pipelines, albeit with computational overhead and limitations tied to the training volume.

Abstract

Visual localization techniques rely upon some underlying scene representation to localize against. These representations can be explicit such as 3D SFM map or implicit, such as a neural network that learns to encode the scene. The former requires sparse feature extractors and matchers to build the scene representation. The latter might lack geometric grounding not capturing the 3D structure of the scene well enough. This paper proposes to jointly learn the scene representation along with a 3D dense feature field and a 2D feature extractor whose outputs are embedded in the same metric space. Through a contrastive framework we align this volumetric field with the image-based extractor and regularize the latter with a ranking loss from learned surface information. We learn the underlying geometry of the scene with an implicit field through volumetric rendering and design our feature field to leverage intermediate geometric information encoded in the implicit field. The resulting features are discriminative and robust to viewpoint change while maintaining rich encoded information. Visual localization is then achieved by aligning the image-based features and the rendered volumetric features. We show the effectiveness of our approach on real-world scenes, demonstrating that our approach outperforms prior and concurrent work on leveraging implicit scene representations for localization.

Figures (8)

• Figure 1: Given an initial pose $R_0,t_0$, to estimate the pose of the query image, we align features from a query image to volumetric features which can be rendered at every position.
• Figure 2: We jointly train an image encoder, a feature and a geometry field. The geometry field is trained through differentiable rendering with photometric and depth information. The feature field is trained by rendering features using the SDF from the geometry field and queried volumetric features. The rendered features are aligned with the image-based feature through a contrastive loss and a prototypical loss.
• Figure 3: Pairs of coarse feature map with $L_{ACL}$ (left) and coarse feature map without $L_{ACL}$ (right) for the same images.
• Figure 4: From left to right, query image, rendered image, rendered depth, the PCA visualisations of encoded/rendered coarse feature maps, PCA visualisations of encoded/rendered fine feature maps.
• Figure 5: Pose refinement on 7-scenes SFM pGT with varying number of rays sampled per iteration and initial learning rate.