Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

One2Any: One-Reference 6D Pose Estimation for Any Object

Mengya Liu, Siyuan Li, Ajad Chhatkuli, Prune Truong, Luc Van Gool, Federico Tombari

TL;DR

One2Any addresses robust 6D pose estimation for unseen objects using only a single RGB-D reference by learning a Reference Object Pose Embedding (ROPE) and predicting a Reference Object Coordinate (ROC) map for a query image. Pose is recovered in the reference camera frame by transforming the query point cloud with the predicted ROC and applying the Umeyama alignment to compute $[\mathbf{R}|\mathbf{t}]$. The approach leverages a latent diffusion-inspired conditioning network with cross-attention (ROPE guiding an OPD U-Net) and a pre-trained VQVAE for query features, enabling fast, single-shot pose estimation without CAD models or multi-view data, while generalizing well to novel objects and occlusions. Quantitative results on Real275, Toyota-Light, LINEMOD, YCB-Video, and other benchmarks show state-of-the-art speed (≈0.09 s/frame) and competitive accuracy, often rivaling multi-view or CAD-based methods. Overall, the method offers scalable, real-time pose estimation for unseen objects with broad applicability to robotics and augmented reality.

Abstract

6D object pose estimation remains challenging for many applications due to dependencies on complete 3D models, multi-view images, or training limited to specific object categories. These requirements make generalization to novel objects difficult for which neither 3D models nor multi-view images may be available. To address this, we propose a novel method One2Any that estimates the relative 6-degrees of freedom (DOF) object pose using only a single reference-single query RGB-D image, without prior knowledge of its 3D model, multi-view data, or category constraints. We treat object pose estimation as an encoding-decoding process, first, we obtain a comprehensive Reference Object Pose Embedding (ROPE) that encodes an object shape, orientation, and texture from a single reference view. Using this embedding, a U-Net-based pose decoding module produces Reference Object Coordinate (ROC) for new views, enabling fast and accurate pose estimation. This simple encoding-decoding framework allows our model to be trained on any pair-wise pose data, enabling large-scale training and demonstrating great scalability. Experiments on multiple benchmark datasets demonstrate that our model generalizes well to novel objects, achieving state-of-the-art accuracy and robustness even rivaling methods that require multi-view or CAD inputs, at a fraction of compute.

One2Any: One-Reference 6D Pose Estimation for Any Object

TL;DR

One2Any addresses robust 6D pose estimation for unseen objects using only a single RGB-D reference by learning a Reference Object Pose Embedding (ROPE) and predicting a Reference Object Coordinate (ROC) map for a query image. Pose is recovered in the reference camera frame by transforming the query point cloud with the predicted ROC and applying the Umeyama alignment to compute . The approach leverages a latent diffusion-inspired conditioning network with cross-attention (ROPE guiding an OPD U-Net) and a pre-trained VQVAE for query features, enabling fast, single-shot pose estimation without CAD models or multi-view data, while generalizing well to novel objects and occlusions. Quantitative results on Real275, Toyota-Light, LINEMOD, YCB-Video, and other benchmarks show state-of-the-art speed (≈0.09 s/frame) and competitive accuracy, often rivaling multi-view or CAD-based methods. Overall, the method offers scalable, real-time pose estimation for unseen objects with broad applicability to robotics and augmented reality.

Abstract

6D object pose estimation remains challenging for many applications due to dependencies on complete 3D models, multi-view images, or training limited to specific object categories. These requirements make generalization to novel objects difficult for which neither 3D models nor multi-view images may be available. To address this, we propose a novel method One2Any that estimates the relative 6-degrees of freedom (DOF) object pose using only a single reference-single query RGB-D image, without prior knowledge of its 3D model, multi-view data, or category constraints. We treat object pose estimation as an encoding-decoding process, first, we obtain a comprehensive Reference Object Pose Embedding (ROPE) that encodes an object shape, orientation, and texture from a single reference view. Using this embedding, a U-Net-based pose decoding module produces Reference Object Coordinate (ROC) for new views, enabling fast and accurate pose estimation. This simple encoding-decoding framework allows our model to be trained on any pair-wise pose data, enabling large-scale training and demonstrating great scalability. Experiments on multiple benchmark datasets demonstrate that our model generalizes well to novel objects, achieving state-of-the-art accuracy and robustness even rivaling methods that require multi-view or CAD inputs, at a fraction of compute.
Paper Structure (18 sections, 8 equations, 4 figures, 6 tables)

This paper contains 18 sections, 8 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Overview
  5. Reference Object Coordinate (ROC)
  6. Reference Object Pose Embedding
  7. Object Pose Decoding with ROPE
  8. Pose Estimation from ROC Map
  9. Experimental Results
  10. Training Details
  11. Dataset
  12. Metrics
  13. Evaluation Procedure
  14. Pose Estimation Performance
  15. Pose Tracking Performance
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Given a single RGB-D image as a reference for an unseen object, our method estimates the pose of the object in a given query image, relative to the reference. The method first predicts a Reference Object Pose Embedding (ROPE) that encodes the object's texture, shape, and pose priors. During inference, each query RGB image is processed through a decoder to predict the Reference Object Coordinate (ROC) map and estimate the relative pose to the reference image. This approach effectively handles large viewpoint changes.
  • Figure 2: Network architecture. The network takes a reference RGB-D image as input and learns a Reference Object Pose Embedding (ROPE) through a Reference Object Encoder (ROE). This embedding is subsequently integrated with the query feature map, which is extracted using a pre-trained VQVAE model rombach2022high with the query RGB image as input. We use the U-Net architecture for effective integrate the ROPE with the query feature with cross-attentions layers. The decoder is trained to predict the ROC map. The final pose estimation is computed using the Umeyama algorithm umeyama1991least.
  • Figure 3: ROC representations given a reference RGB-D image and a query RGB-D image. The ROC space is initially defined by the reference frame, using the camera intrinsics $\mathbf{K}$ and the scaling matrix $\mathbf{S}$ to a normalized space. The query image is subsequently aligned to this space using the relative pose $[\mathbf{R} | \mathbf{t}]$ and the scale matrix $\mathbf{S}$. The ROC map is generated by mapping points in the ROC space to their corresponding 2D pixel locations and encoding the point positions as RGB values.
  • Figure 4: Qualitative results on YCB-Video xiang2017posecnn and LINEMOD hinterstoisser2011multimodal datasets. Predicted poses are displayed in green and ground-truth poses are in pink. We present FoundationPose wen2024foundationpose with the generated CAD models from the reference image (the top is the view close to the reference image, and the bottom is the view close to the query image), and we display Oryon corsetti2024open with the predicted correspondences. For our method, we also show the generated ROC map (bottom) compared with the GT ROC map (top).