Open-vocabulary object 6D pose estimation

TL;DR

This work introduces the new setting of open-vocabulary object 6D pose estimation, in which a textual prompt is used to specify the object of interest, and introduces a novel approach that leverages a Vision-Language Model to segment the object of interest from the scenes and to esti-mate its relative 6D pose.

Abstract

We introduce the new setting of open-vocabulary object 6D pose estimation, in which a textual prompt is used to specify the object of interest. In contrast to existing approaches, in our setting (i) the object of interest is specified solely through the textual prompt, (ii) no object model (e.g., CAD or video sequence) is required at inference, and (iii) the object is imaged from two RGBD viewpoints of different scenes. To operate in this setting, we introduce a novel approach that leverages a Vision-Language Model to segment the object of interest from the scenes and to estimate its relative 6D pose. The key of our approach is a carefully devised strategy to fuse object-level information provided by the prompt with local image features, resulting in a feature space that can generalize to novel concepts. We validate our approach on a new benchmark based on two popular datasets, REAL275 and Toyota-Light, which collectively encompass 34 object instances appearing in four thousand image pairs. The results demonstrate that our approach outperforms both a well-established hand-crafted method and a recent deep learning-based baseline in estimating the relative 6D pose of objects in different scenes. Code and dataset are available at https://jcorsetti.github.io/oryon.

Figures (5)

• Figure 1: Our open-vocabulary setting enables the estimation of the pose of an object captured in two distinct scenes. State-of-the-art generalizable 6D pose estimation methods from RGB on RGBD images typically depend on the object CAD model ove6d or a video sequence of the object at test time gen6donepose as a reference ($A$) to compute the object pose in the query image ($Q$). In contrast, our method uses a textual prompt to guide the pose estimation process, and requires a single view as reference.
• Figure 2: The training pipeline of Oryon can be separated in four stages. In the first stage, the pair of images $A$, $Q$ are encoded by the CLIP image encoder $\phi_V$, while the prompt $T$ is encoded by the CLIP text encoder $\phi_T$. The guidance network $\phi_G$ is used to produce a rich visual representation which is used in the next stages. The resulting visual feature maps $\mathbf{E}^{A}$, $\mathbf{E}^{Q}$ and text features $\mathbf{e}^T$ are used in the second stage in the fusion module $\phi_{TV}$. This outputs a pair of cost features $\mathbf{C}^{A}$,$\mathbf{C}^{Q}$ which in the third stage are upsampled to the final feature maps $\mathbf{F}^{A}$, $\mathbf{F}^{Q}$. The same feature maps are fed to a segmentation head to obtain the predicted masks $\mathbf{M}^{A}$, $\mathbf{M}^{Q}$. At train time, $\mathbf{F}^{A}$, $\mathbf{F}^{Q}$ are optimized by a contrastive loss, while a dice loss supervises the training of the segmentation masks. At test time, the predicted masks are used to filter $\mathbf{F}^{A}$, $\mathbf{F}^{Q}$, and matches are obtained by nearest neighbor. Finally, the matches are projected back to the 3D domain, and a registration algorithm is used to retrieve the final pose $T_{A \rightarrow{Q}}$.
• Figure 3: Examples of qualitative pose results from the REAL275 nocs dataset. All the results use the segmentation mask predicted by Oryon. We show the object model colored my mapping its 3D coordinates to the RGB space.
• Figure 4: Examples of qualitative pose results from the TOYL toyl dataset. All the results use the segmentation mask predicted by Oryon. We show the object model colored my mapping its 3D coordinates to the RGB space.
• Figure 5: Distribution of AR with mIoU (a) and camera distance between image pairs (b) of our best experiment on REAL275 nocs.