Solving Instance Detection from an Open-World Perspective

TL;DR

This work reframes instance detection as an open-world problem and introduces IDOW, a method that adapts pretrained foundation models using metric learning and targeted data augmentations. By incorporating distractor sampling and NeRF-based novel-view synthesis, IDOW enhances instance-level feature matching between visual references and proposals, achieving substantial gains in both conventional CID and novel instance detection (NID) settings. Extensive experiments on HR-InsDet and RoboTools demonstrate more than 10 AP improvements over prior methods, validating the effectiveness of leveraging open-world data and FM adaptation. The approach has practical implications for robotics and AR/VR, offering stronger, more robust instance localization in unseen environments, albeit with considerations around runtime and NeRF training costs for broader deployment.

Abstract

Instance detection (InsDet) aims to localize specific object instances within a novel scene imagery based on given visual references. Technically, it requires proposal detection to identify all possible object instances, followed by instance-level matching to pinpoint the ones of interest. Its open-world nature supports its broad applications from robotics to AR/VR but also presents significant challenges: methods must generalize to unknown testing data distributions because (1) the testing scene imagery is unseen during training, and (2) there are domain gaps between visual references and detected proposals. Existing methods tackle these challenges by synthesizing diverse training examples or utilizing off-the-shelf foundation models (FMs). However, they only partially capitalize the available open-world information. In contrast, we approach InsDet from an Open-World perspective, introducing our method IDOW. We find that, while pretrained FMs yield high recall in instance detection, they are not specifically optimized for instance-level feature matching. Therefore, we adapt pretrained FMs for improved instance-level matching using open-world data. Our approach incorporates metric learning along with novel data augmentations, which sample distractors as negative examples and synthesize novel-view instances to enrich the visual references. Extensive experiments demonstrate that our method significantly outperforms prior works, achieving >10 AP over previous results on two recently released challenging benchmark datasets in both conventional and novel instance detection settings.

Figures (11)

• Figure 1: Instance Detection (InsDet) aims to localize specific object instances in novel scene imagery based on given visual references (aka support templates). It is a challenging problem due to its open-world nature: the testing scene images are unseen during training and thus are unknown to InsDet models, and visual references and detected proposals have domain gaps (e.g., due to occlusions and arbitrary lighting conditions in the latter).
• Figure 2: Existing InsDet methods leverage the open-world information in different aspects, such as (a) background image sampling (from the open world) to synthesize training data Georgakis2017SynthesizingTDdwibedi2017cut, (b) object image sampling (from the open world) to learn feature representations li2024voxdet, and (c) foundation model utilization (pretrained in the open world) for proposal detection and instance-level feature matching shen2023high. (d) As FMs are not specifically designed for instance-level feature matching required by InsDet, we propose to adapt them by leveraging rich data sampled from the open world. We gather data from multiple sources: (1) any available visual references of instances in the CID setting, (2) abundant multi-view object images sampled in the open world similar to (b), (3) synthetic data by training NeRF mildenhall2021nerfbarron2023zip to generate novel-view images based on the given instances, (4) distractors by running FMs (esp. SAM kirillov2023segment) on random open-world imagery to generate random object-like proposals (Fig. \ref{['fig:NeRF-vis']}). We use the data above to adapt FM through metric learning. The technical novelty of our work lies in (3) and (4), as well as the design choice of metric learning to adapt FMs for InsDet.
• Figure 3: Visual comparison of InsDet results by different methods in the CID setting on HR-InsDet shen2023high. The testing scene image contains sparse placement of small instances with challenging illumination. We mark the ground-truth and predictions using green and blue boxes, respectively. We attach instance IDs to them to highlight whether the instance recognition is correct compared to the visual references (i.e., the leftmost references). Compared with Cut-Paste-Learn and OTS-FM, our IDOW detects more instances (see orange arrows) with better accuracy (see red arrows). Comparison between OTS-FM and IDOW suggests suggests that adapting the FM DINOv2 yields better features for IDOW to perform much better in front of domain shifts, i.e., challenging illumination conditions in this testing scene image.
• Figure 4: Visual comparison of InsDet results by different methods in the NID setting on RoboTools li2024voxdet. On the cluttered testing scene image, we mark the ground-truth and predictions using green and blue boxes, respectively. We attach instance IDs to them to highlight whether the instance recognition is correct compared to the visual references (i.e., the leftmost references). Compared with VoxDet and OTS-FM, our IDOW detects more instances (see orange arrows) with better accuracy (see red arrows). Comparing OTS-FM and our IDOW suggests that, although having not seen the visual references during adaption in this NID setting, features by our IDOW are a better representation for InsDet than the off-the-shelf foundation model (i.e., DINOv2) used in OTS-FM.
• Figure 5: We use NeRF to synthesize novel-view object images to augment the limited given visual references. In this work, we train per-instance Zip-NeRF barron2023zip. We visualize synthesized images at different angles (in row 2-4) together with the raw visual references (in the first row). The visual references are from HR-InsDet dataset shen2023high, where the QR code is used for camera pose estimation. Overall, we find synthesized images show high visual quality.