RefPose: Leveraging Reference Geometric Correspondences for Accurate 6D Pose Estimation of Unseen Objects

TL;DR

We address 6D pose estimation for unseen objects from monocular RGB by introducing RefPose, which uses a reference image and geometric correspondence to guide both coarse pose estimation and refinement. A coarse stage selects multiple templates via an optical-flow classifier and employs medoid voting with PnP/RANSAC to produce an initial pose, then a geometry-estimation network with a correlation volume-guided attention produces a query geometry $G_q^{pos}$ aligned to a reference geometry $G_r^{pos}$, enabling iterative refinement with a render-and-compare loop. The approach avoids reliance on shape priors learned from predefined object sets and shows state-of-the-art performance on seven BOP datasets with competitive runtime, validated by thorough ablations. This work advances practical unseen-object pose estimation by integrating reference-guided geometry and attention-based fusion, enabling robust pose estimation in challenging real-world scenes.

Abstract

Estimating the 6D pose of unseen objects from monocular RGB images remains a challenging problem, especially due to the lack of prior object-specific knowledge. To tackle this issue, we propose RefPose, an innovative approach to object pose estimation that leverages a reference image and geometric correspondence as guidance. RefPose first predicts an initial pose by using object templates to render the reference image and establish the geometric correspondence needed for the refinement stage. During the refinement stage, RefPose estimates the geometric correspondence of the query based on the generated references and iteratively refines the pose through a render-and-compare approach. To enhance this estimation, we introduce a correlation volume-guided attention mechanism that effectively captures correlations between the query and reference images. Unlike traditional methods that depend on pre-defined object models, RefPose dynamically adapts to new object shapes by leveraging a reference image and geometric correspondence. This results in robust performance across previously unseen objects. Extensive evaluation on the BOP benchmark datasets shows that RefPose achieves state-of-the-art results while maintaining a competitive runtime.

Figures (7)

• Figure 1: Comparison between conventional methods and proposed method (RefPose). In contrast to conventional methods, RefPose leverages a reference image and geometric correspondence generated from the estimated pose in the coarse pose estimation stage to guide the query's geometric correspondence and pose estimation.
• Figure 2: Overview of the RefPose pipeline. Given an input RGB image, the target object is first detected, cropped, and resized to create the query image, $I_{q}$. In Stage 1: Coarse Pose Estimation, a set of templates, $\mathcal{S}$, is selected from the pre-rendered template set, $\mathcal{T}$, to estimate an initial pose, $P_{0}$, for the query object. In Stage 2: Pose Refinement, the query's geometric correspondence, $G_{q}^{pos}$, is estimated using the rendered reference image, $I_{r}$, and geometric correspondence, $G_{r}^{pos}$. The initial pose, $P_{0}$, is iteratively refined by estimating the relative pose, $\Delta P$, between the query and reference. At each iteration, $G_{r}^{pos}$ is re-rendered to align with the updated pose, leading to an accurate final pose estimate.
• Figure 3: Templates selection using a classification network. The classification network scores pre-rendered templates based on how well optical flow can be estimated between each template and the query image. During inference, these scores are used to select the top-$k$ templates. The classifier leverages a frozen feature encoder from a pre-trained optical flow network, with only the classification head trained.
• Figure 4: Warping-based geometry estimation process. The optical flow between each template image $I_{S}$ in the selected set $\mathcal{S}$ and the query image $I_{q}$ is used to warp the corresponding template geometries $G_{S}$, generating candidate geometries for the query, $G_{q}$. A pixel-wise voting scheme refines these candidates, and the resulting 2D-3D correspondences in $G_{q}$ are then applied with PnP/RANSAC to estimate the initial pose $P_{0}$.
• Figure 5: Architecture of the geometry estimation network. The network takes the query image $I_{q}$, reference image $I_{r}$, and reference geometry $G_{r}^{pos}$ as inputs to estimate the query geometry $G_{q}^{pos}$. A correlation volume-guided attention mechanism is applied at each level of the U-Net to effectively integrate these inputs. The geo head and mask head output a low-resolution geometry map and mask, which are then refined through convex upsampling to produce the final high-resolution geometry $G_{q}^{pos}$.