SRPose: Two-view Relative Pose Estimation with Sparse Keypoints

TL;DR

SRPose tackles two-view relative pose estimation for both camera-to-world and object-to-camera scenarios by leveraging sparse keypoints and a geometry-aware neural network. It introduces an intrinsic-calibration position encoder and promptable prior-knowledge-guided attention to implicitly satisfy the epipolar constraint and regressed the pose directly as $[R|t]$ (where $R\in SO(3)$ and $t\in \mathbb{R}^3$). The 6D rotation representation and an object-prompt mechanism enable mask-free object pose estimation and robust generalization to varying image sizes and camera intrinsics. Experiments show state-of-the-art or competitive performance across camera-to-world, object-to-camera, and map-free relocalization tasks, with substantial speedups due to bypassing traditional robust estimators and direct regression.$

Abstract

Two-view pose estimation is essential for map-free visual relocalization and object pose tracking tasks. However, traditional matching methods suffer from time-consuming robust estimators, while deep learning-based pose regressors only cater to camera-to-world pose estimation, lacking generalizability to different image sizes and camera intrinsics. In this paper, we propose SRPose, a sparse keypoint-based framework for two-view relative pose estimation in camera-to-world and object-to-camera scenarios. SRPose consists of a sparse keypoint detector, an intrinsic-calibration position encoder, and promptable prior knowledge-guided attention layers. Given two RGB images of a fixed scene or a moving object, SRPose estimates the relative camera or 6D object pose transformation. Extensive experiments demonstrate that SRPose achieves competitive or superior performance compared to state-of-the-art methods in terms of accuracy and speed, showing generalizability to both scenarios. It is robust to different image sizes and camera intrinsics, and can be deployed with low computing resources.

Figures (11)

• Figure 1: Relative pose estimation by SRPose. Dots drawn in the figures visualize the cross-attention scores of sparse keypoints across the two images, with brighter dots representing higher attention. Camera-to-world: (a), (b), (c) visualize the epipolar lines, representing the connections between the nine corresponding points across two views. Higher attention is shown to the overlap of the scenes. Object-to-Camera: (d), (e), (f) show the relative 6D pose estimation in the query image with only one accessible object prompt in the reference image. Higher attention is shown to the target object. SRPose establishes implicit correspondences.
• Figure 2: Overview. SRPose comprises four main components: 1) The sparse keypoint detector detects keypoints associated with descriptors separately from the two images; 2) The intrinsic-calibration (IC) position encoder modulates the keypoints' coordinates with camera intrinsics, and encodes their position information; 3) Guided by the prior knowledge of keypoint similarities, along with the object prompt, the attention layers establish implicit cross-view correspondences; 4) The regressor estimates relative pose $R, t$ under the constraints of implicit correspondences.
• Figure 3: Overview of the prior knowledge-guided attention layers. Each layer contains a self-attention and a cross-attention module. A similarity matrix $S$ of the keypoint descriptors is utilized as the prior knowledge to readjust the cross-attention scores, guiding more attention to cross-view keypoint pairs with implicit correspondences. The object prompt and the residual connections are omitted in the figure.
• Figure 4: Time consumption comparison on ScanNet dai2017scannet. Regressors, including SRPose, achieve much higher computational efficacy than all matchers.
• Figure 5: Visualization of implicit cross-view correspondences. Dots and lines drawn with brighter colors represent higher cross-view attention scores. The prior knowledge guidance enhances attention to the overlapping areas, removing the irrelevant cross-view connections, and assisting in establishing implicit correspondences.