NOVO: Bridging LLaVA and SAM with Visual-only Prompts for Reasoning Segmentation

TL;DR

NOVO introduces a visual-prompt–driven framework to address reasoning segmentation by bridging vision-language models with the Segment Anything Model (SAM). Rather than feeding text-derived embeddings, NOVO extracts a coarse mask and point prompts from the VLM output and feeds them into SAM, followed by a training-free refinement that improves mask quality and enables instance-level segmentation. A new RISeg benchmark for instance-level reasoning segmentation demonstrates strong performance gains, with state-of-the-art results on ReasonSeg across 7B and 13B backbones and clear improvements in boundary quality due to NOVO Refinement. Ablation studies confirm the complementary roles of mask and point prompts, and the work discusses limitations and avenues for future expansion, including multi-label reasoning segmentation and dataset growth.

Abstract

In this study, we propose NOVO (NO text, Visual-Only prompts), a novel framework that bridges vision-language models (VLMs) and segmentation models through visual-only prompts. Unlike prior approaches that feed text-derived SEG token embeddings into segmentation models, NOVO instead generates a coarse mask and point prompts from the VLM output. These visual prompts are compatible with the Segment Anything Model (SAM), preserving alignment with its pretrained capabilities. To further enhance boundary quality and enable instance-level segmentation, we introduce a training-free refinement module that reduces visual artifacts and improves the quality of segmentation masks. We also present RISeg, a new benchmark comprising 918 images, 2,533 instance-level masks, and diverse reasoning queries to evaluate this task. Experiments demonstrate that NOVO achieves state-of-the-art performance across multiple metrics and model sizes, demonstrating its effectiveness and scalability in reasoning segmentation.

Figures (7)

• Figure 1: Comparison of VLM activation maps, SAM mask logits, and predicted masks from different models. Our method, which uses visual prompts only, produces masks that better align with the semantics of the input query. In contrast, LISA and READ often show a mismatch between VLM activations and SAM mask logits. Best viewed in color.
• Figure 2: Overview of the proposed NOVO. It encodes an input image $\mathbf{x}_{\text{img}}$ and a reasoning text query $\mathbf{x}_{\text{txt}}$ via a VLM to extract the <SEG> token embedding $\mathbf{t}_{\text{seg}}$. Together with image patch embeddings $\mathbf{I}$, it generates the mask prompt $\mathbf{P}_{\text{mask}}$ and point prompt $\mathbf{P}_{\text{point}}$, which are passed as the sole inputs to the NOVO's segmentation module to produce a segmentation mask $\hat{\mathbf{M}}$. The predicted mask can be refined using our proposed method in Sec. \ref{['sec:4.5']}, which not only improves segmentation quality but also enables instance-level mask generation.
• Figure 3: Overview of the NOVO Refinement. Without any additional training, our refinement method effectively combines the initial mask with SAM’s segmentation capability, not only enhancing the overall segmentation quality but also enabling instance-level reasoning segmentation.
• Figure 4: Overview of the RISeg Dataset. (a) Examples from the RISeg dataset, where each image is paired with a reasoning-based text query and multiple ground-truth instance masks (shown with white contours). (b) Distribution of instance mask classes grouped by categories. (c) The number of annotated instances per image.
• Figure 5: Qualitative comparison of NOVO and existing methods on reasoning segmentation tasks. NOVO produces accurate and coherent masks, even in challenging reasoning cases such as ambiguous boundaries and multi-instance scenarios. More qualitative examples are provided in the supplementary materials.