Structure-Aware Correspondence Learning for Relative Pose Estimation

TL;DR

This work tackles object-relative pose estimation for unseen categories by proposing Structure-Aware Correspondence Learning, which jointly learns structure-aware keypoints and structure-aware features for 3D-3D correspondence without explicit feature matching. The method employs learnable keypoint queries with image reconstruction supervision, intra- and inter-image attention with ROPE-based encoding, and 3D coordinate regression for keypoints, followed by a weighted SVD to recover the relative rotation. Across CO3D, Objaverse, and LineMOD, it achieves state-of-the-art results, notably reducing mean angular error by about $6^{\circ}$ on CO3D and improving angular-accuracy metrics by substantial margins, demonstrating strong generalization to unseen objects. The approach offers a practical path toward object-agnostic pose estimation by leveraging object structure rather than dense 2D- or 3D- feature matching, with implications for AR, robotics, and autonomous systems.

Abstract

Relative pose estimation provides a promising way for achieving object-agnostic pose estimation. Despite the success of existing 3D correspondence-based methods, the reliance on explicit feature matching suffers from small overlaps in visible regions and unreliable feature estimation for invisible regions. Inspired by humans' ability to assemble two object parts that have small or no overlapping regions by considering object structure, we propose a novel Structure-Aware Correspondence Learning method for Relative Pose Estimation, which consists of two key modules. First, a structure-aware keypoint extraction module is designed to locate a set of kepoints that can represent the structure of objects with different shapes and appearance, under the guidance of a keypoint based image reconstruction loss. Second, a structure-aware correspondence estimation module is designed to model the intra-image and inter-image relationships between keypoints to extract structure-aware features for correspondence estimation. By jointly leveraging these two modules, the proposed method can naturally estimate 3D-3D correspondences for unseen objects without explicit feature matching for precise relative pose estimation. Experimental results on the CO3D, Objaverse and LineMOD datasets demonstrate that the proposed method significantly outperforms prior methods, i.e., with 5.7°reduction in mean angular error on the CO3D dataset.

Figures (3)

• Figure 1: a) Task depiction. Predict the relative pose $\Delta R$ between the query and reference images. b) 2D correspondence-based methods extract keypoints to conduct 2D-2D matching for pose estimation. c) 3D correspondence-based methods lift 2D features into 3D voxel features and conduct 3D-3D matching for pose estimation. d) Our method bypasses the feature matching and directly regresses 3D correspondences for pose estimation.
• Figure 2: a) Overview of the proposed method. b) Illustration of the Structure-Aware Keypoint Extraction module. We initialize a set of learnable queries that interact with image features to extract keypoints representing the object's structure. And we further employ a reconstructor and $\mathcal{L}_{\text{rec}}$ for supervision of keypoints extraction. c) Illustration of the Structure-Aware Correspondence Estimation module. We employ ROPE and an attention mechanism to extract structure-aware features for 3D correspondence estimation.
• Figure 3: Qualitative comparisons among LoFTR, DVMNet and our method. We visualize the ground truth and predicted arrows. Blue indicates ground truth and green indicates prediction.