OPFormer: Object Pose Estimation leveraging foundation model with geometric encoding

TL;DR

OPFormer tackles 6D pose estimation of unseen objects by unifying object onboarding (CAD-based or NeRF-based) with a transformer-based pose estimator that encodes 3D geometry via NOCS and learns inter-template relations. It leverages a weight-adapted DINOv2 feature extractor and a 3D rotary positional encoding within a cross-attentive encoder-decoder to establish robust 2D-3D correspondences, solved by PnP within RANSAC. The approach achieves a strong balance between accuracy and efficiency on the BOP benchmarks, including both model-based and model-free onboarding scenarios, and demonstrates competitive or state-of-the-art performance with fast inference. The proposed method introduces practical onboarding for unseen objects and a geometry-aware representation that generalizes across object instances, enabling real-time deployment in robotics and AR applications.

Abstract

We introduce a unified, end-to-end framework that seamlessly integrates object detection and pose estimation with a versatile onboarding process. Our pipeline begins with an onboarding stage that generates object representations from either traditional 3D CAD models or, in their absence, by rapidly reconstructing a high-fidelity neural representation (NeRF) from multi-view images. Given a test image, our system first employs the CNOS detector to localize target objects. For each detection, our novel pose estimation module, OPFormer, infers the precise 6D pose. The core of OPFormer is a transformer-based architecture that leverages a foundation model for robust feature extraction. It uniquely learns a comprehensive object representation by jointly encoding multiple template views and enriches these features with explicit 3D geometric priors using Normalized Object Coordinate Space (NOCS). A decoder then establishes robust 2D-3D correspondences to determine the final pose. Evaluated on the challenging BOP benchmarks, our integrated system demonstrates a strong balance between accuracy and efficiency, showcasing its practical applicability in both model-based and model-free scenarios.

Figures (13)

• Figure 1: We illustrate the three major blocks of our pipeline and their respective computational times: (a) object onboarding from a CAD model or a set of images takes less than a minute per object, (b) object detection with CNOS nguyen2023cnos takes less than 0.5 seconds per image, (c) pose estimation with our methodology takes less than 0.05 seconds per instance.
• Figure 2: Our onboarding pipeline functions with either a CAD model or a set of images with accurate pose annotation. Given a set of multi-view images of an object, we train a NeRF using the Instant Neural Graphics Primitives (iNGP) framework. RGB, depth, and NOCS templates are rendered from either CAD model or the trained NeRF. The pose estimation framework subsequently utilizes these templates along with the cropped test image. The RGB templates and test image crop are initially passed through the ViT-L DINOv2 block to extract the features. Template features are then passed to the encoder block, along with NOCS maps, to obtain positionally embedded template representation. The decoder determines relations between template and test image encodings via cross-attention. The feature matching stage establishes 2D-3D correspondences, and estimates 6D object pose by solving the PnP problem.
• Figure 3: Architecture of the weight adapter (WA) module. Multi-layer patch descriptors from a DINOv2 model are dimensionally reduced, then processed by dilated convolutions. The outputs are then summed and normalized to generate the final feature set.
• Figure 4: Visualization of the ground-truth (green) and estimated (magenta) 6D poses is presented on two datasets (from the first row to last): YCB-V and T-LESS. The first column displays the test image with a contour highlight of the projection made from the predicted pose. The subsequent three columns depicts the corresponding 3D views from different viewing angles. The first 3D view is taken from approximately the same viewing angle as the the image. Additional visualization is provided in \ref{['sec:A_visualization']}.
• Figure 5: HANDAL dataset visualization of 2D projection made with the predicted 6D poses. In the left column is visualized contour highlight and in the right the projected masks of the prediction