OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts

TL;DR

OpenWorldSAM tackles open-vocabulary image segmentation by extending SAM2 with a lightweight language adapter that fuses a frozen vision-language encoder (BEiT-3) and a set of learnable tie-breakers and a cross-attention based soft prompt, enabling single-text prompts to disambiguate and segment multiple instances. The method preserves SAM2’s backbone and segmentation capability while adding only about $4.5$ million trainable parameters, achieving strong zero-shot and referring-expression performance across ADE20K, PASCAL Context, ScanNet, and RefCOCOg. Key contributions include (i) positional tie-breakers for multi-instance separation, (ii) soft prompting via cross-attention to ground language in image features, and (iii) a unified, prompt-driven interface for semantic, instance, panoptic, and referring-segmentation, with oracle-prompt evaluation proposed for fair comparison. The results demonstrate state-of-the-art zero-shot open-vocabulary segmentation with much lower trainable-parameter counts and preserve interactive capabilities, pointing to practical deployment in real-world, open-world perception systems.

Abstract

The ability to segment objects based on open-ended language prompts remains a critical challenge, requiring models to ground textual semantics into precise spatial masks while handling diverse and unseen categories. We present OpenWorldSAM, a framework that extends the prompt-driven Segment Anything Model v2 (SAM2) to open-vocabulary scenarios by integrating multi-modal embeddings extracted from a lightweight vision-language model (VLM). Our approach is guided by four key principles: i) Unified prompting: OpenWorldSAM supports a diverse range of prompts, including category-level and sentence-level language descriptions, providing a flexible interface for various segmentation tasks. ii) Efficiency: By freezing the pre-trained components of SAM2 and the VLM, we train only 4.5 million parameters on the COCO-stuff dataset, achieving remarkable resource efficiency. iii) Instance Awareness: We enhance the model's spatial understanding through novel positional tie-breaker embeddings and cross-attention layers, enabling effective segmentation of multiple instances. iv) Generalization: OpenWorldSAM exhibits strong zero-shot capabilities, generalizing well on unseen categories and an open vocabulary of concepts without additional training. Extensive experiments demonstrate that OpenWorldSAM achieves state-of-the-art performance in open-vocabulary semantic, instance, and panoptic segmentation across multiple benchmarks. Code is available at https://github.com/GinnyXiao/OpenWorldSAM.

Figures (9)

• Figure 1: Overview of the proposed framework. The green region highlights the SAM v2 baseline, supporting visual prompts (e.g., boxes, points) for interactive segmentation. Our OpenWorldSAM extension integrates open-vocabulary language understanding, enabling both category-level segmentation across semantic, instance, panoptic tasks and referring expression segmentation.
• Figure 2: OpenWorldSAM achieves new state-of-the-art on six datasets with one suite of parameters.
• Figure 3: (a) SAM takes a visual click and outputs 3 valid masks on the same person (the person, the backpack, and a backpack region) kirillov2023sam. It will not output masks for the person standing next to her. (b) Tie-breakers shift the queries to distinct regions, enabling simultaneous segmentation of all three "zebra" instances. (c) Naïve approach zhang2024evfsam: A single language query for “zebra” causes SAM2 to segment only the most salient instance.
• Figure 4: (a) Preliminaries on the inputs and outputs of the vision and multi-modal encoders. (b) OpenWorldSAM pipeline. (c) Detailed soft prompting Transformer architecture.
• Figure 5: Qualitative comparisons on ADE20K-857. In many cases, (e.g., (c) road, field), X-Decoder predicts semantically related but incorrect labels due to ambiguity in the category list. The final column shows X-Decoder predictions using oracle prompts, which reduces confusion. OpenWorldSAM, conditioned on the correct prompt, produces faithful masks and avoids semantic mismatches. Color maps for each model vary. Please refer to the predicted labels. Best viewed with zoom in. We use two-stage inference for the visualization.