NOCTIS: Novel Object Cyclic Threshold based Instance Segmentation

TL;DR

NOCTIS tackles zero-shot instance segmentation of unseen objects in RGB images without retraining. It fuses Grounded-SAM 2 for proposals with DINOv2 descriptors, introducing a cyclic-threshold patch matching, an aggregated appearance score, and proposal-confidence weighting to improve cross-object discrimination. The method achieves state-of-the-art mean AP on seven core BOP 2023 datasets for model-based 2D segmentation of unseen objects, surpassing RGB and RGB-D baselines and the strongest published RGB-based approaches. This work demonstrates that carefully designed scoring and robust foundation-model usage can outperform depth-enhanced methods in novel-object segmentation, offering a practical, training-free baseline for real-world deployment.

Abstract

Instance segmentation of novel objects instances in RGB images, given some example images for each object, is a well known problem in computer vision. Designing a model general enough to be employed for all kinds of novel objects without (re-) training has proven to be a difficult task. To handle this, we present a new training-free framework, called: Novel Object Cyclic Threshold based Instance Segmentation (NOCTIS). NOCTIS integrates two pre-trained models: Grounded-SAM 2 for object proposals with precise bounding boxes and corresponding segmentation masks; and DINOv2 for robust class and patch embeddings, due to its zero-shot capabilities. Internally, the proposal-object matching is realized by determining an object matching score based on the similarity of the class embeddings and the average maximum similarity of the patch embeddings with a new cyclic thresholding (CT) mechanism that mitigates unstable matches caused by repetitive textures or visually similar patterns. Beyond CT, NOCTIS introduces: (i) an appearance score that is unaffected by object selection bias; (ii) the usage of the average confidence of the proposals' bounding box and mask as a scoring component; and (iii) an RGB-only pipeline that performs even better than RGB-D ones. We empirically show that NOCTIS, without further training/fine tuning, outperforms the best RGB and RGB-D methods regarding the mean AP score on the seven core datasets of the BOP 2023 challenge for the "Model-based 2D segmentation of unseen objects" task.

Figures (3)

• Figure 1: The three NOCTIS stages: onboarding stage, represents each object via descriptors from templates ( \ref{['sec:method:onboard']}); proposal stage ( \ref{['sec:method:prop']}), where proposals (masks) and their descriptors from the query RGB image are generated; lastly, in the matching stage, object labels and confidences are assigned to each proposal based on their descriptors ( \ref{['sec:method:match']}).
• Figure 2: A general representation of the cyclic distance of patch $s$ through $t$ and $u$. Each image is divided into a $8 \times 8$ grid. Starting from patch $s$ in image ${\bm{A}}$, the most cosine similar patch in ${\bm{B}}$ is $t$. Vice versa, starting from $t$, its best match in ${\bm{A}}$ is $u$. The patch $u$ is the cyclic/round-trip patch of $s$, their euclidean distance is called the cyclic distance.
• Figure 3: Qualitative assessment of some segmentation results using CNOS, SAM-6D, NIDS-Net, and NOCTIS on YCB-V and T-LESS. The image addresses the strengths and limitations of these methods. The red arrows indicate errors in the segmentation masks and/or classifications of the proposals. For better visualization purposes, $\delta_{\mathit{conf}} = 0.5$ was used.