OSCAR: Open-Set CAD Retrieval from a Language Prompt and a Single Image

TL;DR

OSCAR addresses open-set 6D object pose estimation when exact CAD models are unavailable by proposing a training-free CAD retrieval framework that uses a language prompt and a single image. It onboarding renders and captions all CAD models, then performs a two-stage retrieval that first filters by text similarity with CLIP and then refines by image similarity with DINOv2, conditioned on a GroundedSAM ROI. The approach achieves state-of-the-art results on MI3DOR and significantly improves pose estimation when using retrieved CAD models with MegaPose, while offering substantial time savings over on-the-fly reconstructions. This work enables scalable, open-set object model sourcing in dynamic environments, boosting robustness and efficiency for robotics and AR applications.

Abstract

6D object pose estimation plays a crucial role in scene understanding for applications such as robotics and augmented reality. To support the needs of ever-changing object sets in such context, modern zero-shot object pose estimators were developed to not require object-specific training but only rely on CAD models. Such models are hard to obtain once deployed, and a continuously changing and growing set of objects makes it harder to reliably identify the instance model of interest. To address this challenge, we introduce an Open-Set CAD Retrieval from a Language Prompt and a Single Image (OSCAR), a novel training-free method that retrieves a matching object model from an unlabeled 3D object database. During onboarding, OSCAR generates multi-view renderings of database models and annotates them with descriptive captions using an image captioning model. At inference, GroundedSAM detects the queried object in the input image, and multi-modal embeddings are computed for both the Region-of-Interest and the database captions. OSCAR employs a two-stage retrieval: text-based filtering using CLIP identifies candidate models, followed by image-based refinement using DINOv2 to select the most visually similar object. In our experiments we demonstrate that OSCAR outperforms all state-of-the-art methods on the cross-domain 3D model retrieval benchmark MI3DOR. Furthermore, we demonstrate OSCAR's direct applicability in automating object model sourcing for 6D object pose estimation. We propose using the most similar object model for pose estimation if the exact instance is not available and show that OSCAR achieves an average precision of 90.48\% during object retrieval on the YCB-V object dataset. Moreover, we demonstrate that the most similar object model can be utilized for pose estimation using Megapose achieving better results than a reconstruction-based approach.

Figures (7)

• Figure 1: Overview: We propose OSCAR to retrieve a 3D object model from an unlabeled database of 3D objects with a single image input and a language prompt.
• Figure 2: Concept overview: During the onboarding stage, each 3D model is rendered from multiple views and automatically captioned to enable language-guided retrieval without manual labels. During the inference stage, a Region-of-Interest (ROI) is generated from an RGB image and a language prompt using GroundedSAM. First, CLIP embeddings of the ROI and model captions are compared via cosine similarity and filtered according to a threshold. The DINOv2 embeddings are computed for these candidate images. Based on the cosine similarity, the object model with the highest similarity is selected.
• Figure 3: Visualization of object meshes: Ground-truth object models (top), most frequently confused CAD model retrieved by OSCAR (middle), and Nerfacto-reconstructed models (bottom).
• Figure 4: Comparison of pose estimation results across three approaches: (a) ground-truth object models, (b) CAD models retrieved by OSCAR, and (c) reconstructed 3D models generated with Nerfacto.
• Figure 6: Example image (top) and table (bottom) comparing four prompting strategies. The table lists the prompt type, prompt text, and an example output.