MUSE: Model-based Uncertainty-aware Similarity Estimation for zero-shot 2D Object Detection and Segmentation

TL;DR

MUSE addresses zero-shot 2D object detection and segmentation without any training by integrating class and patch embeddings learned from vision foundation models through a GeM-based fusion, and by using a joint absolute-relative similarity along with an uncertainty-aware objectness prior. It introduces a four-stage pipeline (onboarding, proposal, embedding, matching) tailored for model-based recognition, and demonstrates state-of-the-art performance on the BOP 2025 benchmark across Classic-Core, H3, and Industrial tracks. Key contributions include (i) a GeM-based feature integration scheme, (ii) a joint similarity score that blends absolute and relative cues, and (iii) a Bayesian-inspired objectness prior that conditions the final predictions on proposal reliability. The results show robust and scalable zero-shot detection and segmentation in cluttered and industrial settings, highlighting MUSE's generalizability and practical impact for real-world deployment.

Abstract

In this work, we introduce MUSE (Model-based Uncertainty-aware Similarity Estimation), a training-free framework designed for model-based zero-shot 2D object detection and segmentation. MUSE leverages 2D multi-view templates rendered from 3D unseen objects and 2D object proposals extracted from input query images. In the embedding stage, it integrates class and patch embeddings, where the patch embeddings are normalized using generalized mean pooling (GeM) to capture both global and local representations efficiently. During the matching stage, MUSE employs a joint similarity metric that combines absolute and relative similarity scores, enhancing the robustness of matching under challenging scenarios. Finally, the similarity score is refined through an uncertainty-aware object prior that adjusts for proposal reliability. Without any additional training or fine-tuning, MUSE achieves state-of-the-art performance on the BOP Challenge 2025, ranking first across the Classic Core, H3, and Industrial tracks. These results demonstrate that MUSE offers a powerful and generalizable framework for zero-shot 2D object detection and segmentation.

Figures (6)

• Figure 1: Comparison of detection performance on the BOP Classic Core benchmark. Each axis corresponds to a dataset, and values are mAP scores. It is demonstrated that MUSE (dotted red) consistently outperforms other methods in all datasets.
• Figure 2: Overview of the proposed framework. Templates and proposals are obtained in the onboarding stage and the proposal stage, respectively. Then, DINOv2 oquab2023dinov2 yields class and patch embeddings from both templates and proposals. To integrate class and patch embedding into similarity metric, we introduce a generalized mean pooling (GeM). Finally, we further modify the similarity metric using both a joint similarity score scheme and an uncertainty-aware objectness prior, which yields state-of-the-art object detection results.
• Figure 3: Visualization of the top-1 retrieved templates with respect to embedding types at the LM-O dataset. Each column shows the retrieval templates for class embeddings (cls), patch embeddings (patch), and our integrated embeddings between cls and patch, respectively. It is demonstrated that the proposed integrated embedding method shows both class-level and geometry-level consistencies.
• Figure 4: Effect of the joint similarity score. The joint similarity score efficiently predicts objects even in the cluttered scene (see top row). Also, visually similar objects (obj6 vs. obj7) are clearly separated (see bottom row).
• Figure 5: Effect of the uncertainty-aware objectness prior. False detections are effectively removed due to the objectness prior. With the Bayesian formulation (Eq. \ref{['eq:final_score']}), the rescaled objectness prior yields more accurate detections.