High-resolution open-vocabulary object 6D pose estimation

TL;DR

Horyon tackles open-vocabulary 6D pose estimation by describing an unseen object with natural language and localizing it in two RGBD scenes. It employs a Vision-Language backbone (DINO + BERT) with cross-attention fusion to produce high-resolution, multi-scale features, enabling pixel-level matching and registration to recover the 6D pose $T_{A \rightarrow Q}$. Key innovations include using GroundingDino to crop objects, a token-wise textual representation, and a high-resolution decoder with guidance features to preserve fine-grained appearance details during matching. Evaluated on REAL275, Toyota-Light, Linemod, and YCB-Video, Horyon achieves state-of-the-art Average Recall, outperforming prior open-vocabulary methods and demonstrating strong generalization to unseen objects and robustness to prompt noise. Limitations include reliance on depth maps and intrinsic parameters, suggesting future work toward monocular depth estimation and richer prompts.

Abstract

The generalisation to unseen objects in the 6D pose estimation task is very challenging. While Vision-Language Models (VLMs) enable using natural language descriptions to support 6D pose estimation of unseen objects, these solutions underperform compared to model-based methods. In this work we present Horyon, an open-vocabulary VLM-based architecture that addresses relative pose estimation between two scenes of an unseen object, described by a textual prompt only. We use the textual prompt to identify the unseen object in the scenes and then obtain high-resolution multi-scale features. These features are used to extract cross-scene matches for registration. We evaluate our model on a benchmark with a large variety of unseen objects across four datasets, namely REAL275, Toyota-Light, Linemod, and YCB-Video. Our method achieves state-of-the-art performance on all datasets, outperforming by 12.6 in Average Recall the previous best-performing approach.

Figures (3)

• Figure 1: The main modules of our proposed method, Horyon. (a) Overview of a processing branch: Horyon first crops the object of interest from the scene given a textual prompt, and subsequently extracts visual and textual features with DINO and BERT, respectively from the image crop and the prompt. The fusion module $\phi_{TV}$ outputs a multimodal representation of the scene, which is upsampled by a decoder $\phi_D$. At this stage, skip connections from the image encoder $\phi_V$ are used to enrich the final representation. The output features $\mathbf{F}^{}$ are used to obtain the object segmentation mask $\mathbf{M}^{}$. (b) Optimisation procedure. $\mathbf{F}^{A}$, $\mathbf{F}^{Q}$ are optimised by a hardest contrastive loss which uses ground-truth matches as supervision, while the segmentation is supervised by a Dice loss. (c) Test procedure. The predicted masks are used to filter $\mathbf{F}^{A}$, $\mathbf{F}^{Q}$, and matches are obtained by nearest neighbor. Finally, the matches are backprojected in 3D, and a registration algorithm is used to retrieve the final pose $T_{A \rightarrow{Q}}$.
• Figure 2: (a) Overview of a layer of the fusion module $\phi_{TV}$. Note that the left-side output of the module is not used in the last layer. (b) Architecture of the decoder $\phi_D$. $\mathbf{E}^{}_{1}$,$\mathbf{E}^{}_{2}$,$\mathbf{E}^{}_{3}$: visual features obtained from $\phi_V$; Transp.Conv.: transposed 2D convolution; Group.Conv.: group composed by two blocks, each contains a 2D convolution, a group normalisation and a ReLU; $[ \cdot | \cdot ]$ denotes feature concatenation.
• Figure 3: Sample pose results from REAL275 nocs (a), Toyota-Light toyl (b), Linemod lm (c) and YCB-Video ycbv (d). All the results use crop from GroundingDino liu2023groundingdino and segmentation mask predicted by Horyon. We show the object model coloured by mapping its 3D coordinates to the RGB space. Query images are darkened to highlight the object poses.