Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactively

TL;DR

The Open-Vocabulary SAM is introduced, a SAM-inspired model designed for simultaneous interactive segmentation and recognition, leveraging two unique knowledge transfer modules: SAM2CLIP and CLIP2SAM.

Abstract

The CLIP and Segment Anything Model (SAM) are remarkable vision foundation models (VFMs). SAM excels in segmentation tasks across diverse domains, whereas CLIP is renowned for its zero-shot recognition capabilities. This paper presents an in-depth exploration of integrating these two models into a unified framework. Specifically, we introduce the Open-Vocabulary SAM, a SAM-inspired model designed for simultaneous interactive segmentation and recognition, leveraging two unique knowledge transfer modules: SAM2CLIP and CLIP2SAM. The former adapts SAM's knowledge into the CLIP via distillation and learnable transformer adapters, while the latter transfers CLIP knowledge into SAM, enhancing its recognition capabilities. Extensive experiments on various datasets and detectors show the effectiveness of Open-Vocabulary SAM in both segmentation and recognition tasks, significantly outperforming the naïve baselines of simply combining SAM and CLIP. Furthermore, aided with image classification data training, our method can segment and recognize approximately 22,000 classes.

Figures (10)

• Figure 1: Open-Vocabulary SAM not only can segment anything with prompts just like SAM but also has the capability of recognition in the real world, like CLIP. With drastically lower computational cost, Open-Vocabulary SAM has a higher recognition performance than directly combining SAM and CLIP with image or feature cropping (measured on the COCO open vocabulary benchmark).
• Figure 2: Comparison of two simple SAM-CLIP combination baselines (a) and (b), and our proposed single encoder architecture (c). The adapters for (a) and (b) are optional and can be replaced with various designs (please refer to Sec. \ref{['sec:main_results']} for details). Note that, in our method, the SAM encoder will be discarded during inference.
• Figure 3: Illustration of Open-Vocabulary SAM. For training, the SAM encoder is as a teacher network, while SAM2CLIP plays the role of a student network and aligns the knowledge of SAM into CLIP. The CLIP2SAM transfers the CLIP knowledge to the SAM decoder and performs joint segmentation and classification for close-set and open vocabulary settings.
• Figure 4: Visualization Comparison. We compare the mask and classification results of the image-crop baseline (SAM + CLIP) and Open-Vocabulary SAM (OV SAM). The predicted labels are presented on the mask.
• Figure 5: Qualitative results of Open-Vocabulary SAM. In the visualization, boxes refer to box prompts (up), and the red stars refer to the point prompts (middle). The masks can be merged into a segmentation map (bottom). We show the mask and labels generated by the proposed Open-Vocabulary SAM. Our method can segment and recognize open vocabulary objects in diverse scenes identified by prompts.