LiftFeat: 3D Geometry-Aware Local Feature Matching

TL;DR

This paper addresses robust local feature matching in challenging conditions by introducing LiftFeat, a lightweight architecture that augments 2D descriptors with 3D geometric cues. It learns surface normals via supervision from monocular depth estimates (Depth Anything v2) and fuses these normals with 2D descriptors through a 3D Geometry-aware Feature Lifting module, followed by self-attention-based refinement. The authors formulate a multi-task training regime with keypoint, normal, and descriptor losses and demonstrate improved performance on relative pose estimation, homography estimation, and visual localization, including nighttime scenarios, while achieving edge-friendly runtimes (~7.4 ms). Overall, LiftFeat offers a practical, geometry-informed enhancement to local feature matching that preserves efficiency for robotic applications and edge devices.

Abstract

Robust and efficient local feature matching plays a crucial role in applications such as SLAM and visual localization for robotics. Despite great progress, it is still very challenging to extract robust and discriminative visual features in scenarios with drastic lighting changes, low texture areas, or repetitive patterns. In this paper, we propose a new lightweight network called \textit{LiftFeat}, which lifts the robustness of raw descriptor by aggregating 3D geometric feature. Specifically, we first adopt a pre-trained monocular depth estimation model to generate pseudo surface normal label, supervising the extraction of 3D geometric feature in terms of predicted surface normal. We then design a 3D geometry-aware feature lifting module to fuse surface normal feature with raw 2D descriptor feature. Integrating such 3D geometric feature enhances the discriminative ability of 2D feature description in extreme conditions. Extensive experimental results on relative pose estimation, homography estimation, and visual localization tasks, demonstrate that our LiftFeat outperforms some lightweight state-of-the-art methods. Code will be released at : https://github.com/lyp-deeplearning/LiftFeat.

Figures (3)

• Figure 1: Feature matching of applying 2D visual cues and integrating 3D geometric cues in a low texture scene. Green lines: correct matches; Red lines: incorrect matches. Histograms: distribution of descriptor features. (a) Result of using SuperPoint detone2018superpoint. (b) 3D geometric Normal Map. (c) Result of using our LiftFeat. Incorporating 3D information enhances the distinctiveness of the raw 2D descriptors.
• Figure 2: Overview of the proposed LiftFeat. Given an input image $I$, the feature extraction module outputs keypoint map, description map, and normal map through separate multi-task heads. During the training phase, we use the predicted depth map from the Depth Anything v2 yang2024depth to obtain pseudo normal label as a supervisory signal to assist in learning 3D geometric features. Finally, the 3D geometric-aware feature lifting module fuses the 2D and 3D features.
• Figure 3: Qualitative matching results. We conduct tests in both indoor and outdoor scenes. The results demonstrate that our proposed LiftFeat maintains robust matching performance under extreme conditions, such as lighting variations (top), low texture (middle), and repetitive pattern (bottom) scenarios. Green lines: correct matches; Red lines: incorrect matches.