Show or Tell? Effectively prompting Vision-Language Models for semantic segmentation

TL;DR

The paper addresses whether large vision-language models can be guided to perform semantic segmentation through prompting alone. It introduces FPSS, a few-shot prompted segmentation paradigm, and systematically compares text and visual prompts on the MESS dataset, finding a ~30% IoU gap to domain-specific models and strong complementarity between TP and VP. Building on this, it proposes PromptMatcher, a training-free framework that fuses text-based and visual prompts with a verification step, achieving state-of-the-art performance on few-shot prompted segmentation and surpassing single-modality baselines. The work highlights the practical potential of multimodal prompting for segmentation and motivates future research into robust, prompt-driven foundation-model pipelines.

Abstract

Large Vision-Language Models (VLMs) are increasingly being regarded as foundation models that can be instructed to solve diverse tasks by prompting, without task-specific training. We examine the seemingly obvious question: how to effectively prompt VLMs for semantic segmentation. To that end, we systematically evaluate the segmentation performance of several recent models guided by either text or visual prompts on the out-of-distribution MESS dataset collection. We introduce a scalable prompting scheme, few-shot prompted semantic segmentation, inspired by open-vocabulary segmentation and few-shot learning. It turns out that VLMs lag far behind specialist models trained for a specific segmentation task, by about 30% on average on the Intersection-over-Union metric. Moreover, we find that text prompts and visual prompts are complementary: each one of the two modes fails on many examples that the other one can solve. Our analysis suggests that being able to anticipate the most effective prompt modality can lead to a 11% improvement in performance. Motivated by our findings, we propose PromptMatcher, a remarkably simple training-free baseline that combines both text and visual prompts, achieving state-of-the-art results outperforming the best text-prompted VLM by 2.5%, and the top visual-prompted VLM by 3.5% on few-shot prompted semantic segmentation.

Figures (6)

• Figure 1: The FPSS task involves providing a VLM with visual (image + corresponding mask) and text prompt. The goal is for the model to make predictions on new target images.
• Figure 2: Qualitative analysis of the results of LISA and SoftMatcher+ compared to ground truth. The first four columns display images selected according to biggest difference of IoU between VP and TP as per Table \ref{['tab:performance_difference']}. The last column displays the Tool class.
• Figure 3: PromptMatcher framework: The left section illustrates the mask generation process using visual and text prompts, while the right section shows the verification module which discards inaccurate predictions.
• Figure 4: Qualitative examples selected from the most challenging classes of FoodSeg103.
• Figure 5: Qualitative analysis on examples of challenging classes for Text Prompting.