Vision-Language Model for Object Detection and Segmentation: A Review and Evaluation

TL;DR

The paper addresses open-vocabulary perception by evaluating Vision-Language Models (VLMs) as foundation models for detection and segmentation. It systematically benchmarks a wide spectrum of methods—spanning large-scale pretraining, distillation, pseudo-labeling, multi-task learning, prompts, and LLM augmentation—across eight detection and eight segmentation tasks, including domain adaptation, few-shot, robustness, and dense-object scenarios. Key findings show that large-scale pretraining generally strengthens OV capabilities, visual fine-tuning improves base-category accuracy, and prompt-based strategies offer domain-dependent gains, with transformer-based detectors like GroundingDINO delivering strong cross-modal fusion. The study provides detailed insights into task characteristics, model architectures, and training regimes, offering practical guidance for designing future VLM-based perception systems and establishing robust evaluation baselines across diverse datasets.

Abstract

Vision-Language Model (VLM) have gained widespread adoption in Open-Vocabulary (OV) object detection and segmentation tasks. Despite they have shown promise on OV-related tasks, their effectiveness in conventional vision tasks has thus far been unevaluated. In this work, we present the systematic review of VLM-based detection and segmentation, view VLM as the foundational model and conduct comprehensive evaluations across multiple downstream tasks for the first time: 1) The evaluation spans eight detection scenarios (closed-set detection, domain adaptation, crowded objects, etc.) and eight segmentation scenarios (few-shot, open-world, small object, etc.), revealing distinct performance advantages and limitations of various VLM architectures across tasks. 2) As for detection tasks, we evaluate VLMs under three finetuning granularities: \textit{zero prediction}, \textit{visual fine-tuning}, and \textit{text prompt}, and further analyze how different finetuning strategies impact performance under varied task. 3) Based on empirical findings, we provide in-depth analysis of the correlations between task characteristics, model architectures, and training methodologies, offering insights for future VLM design. 4) We believe that this work shall be valuable to the pattern recognition experts working in the fields of computer vision, multimodal learning, and vision foundation models by introducing them to the problem, and familiarizing them with the current status of the progress while providing promising directions for future research. A project associated with this review and evaluation has been created at https://github.com/better-chao/perceptual_abilities_evaluation.

Figures (8)

• Figure 1: Illustration of Evaluation Framework for Vision-Language Models in Detection and Segmentation Tasks. For detection VLM models, we conduct comprehensive evaluations across: Traditional Closed-Set, Open Vocabulary, Fine-Grained Perception, Vocabulary Fine-Grained Perception, Few-Shot, Robustness, Domain-Related, Dense Object tasks. For segmentation VLM models, we perform systematic evaluation on Open Vocabulary, Multi Domain, Fine-Grained, Few-Shot, Robustness, Zero-Shot, Dense object, Small object tasks.
• Figure 2: Illustration of three granular fine-tuning strategies for visual-language detection models. (a) Zero Prediction directly evaluates the VLM on downstream tasks without fine-tuning. (b) Visual Fine-tuning adapts the VLM's visual branch on downstream data before evaluation, and (c) Text Prompt optimizes only the text prompts with downstream data prior to evaluation.
• Figure 3: The timeline of VLM-based detection methods.
• Figure 4: Different types of VLM-based Detection Methods. Large-scale Pretraining Based Methods are trained on large-scale dataset to improve zero-shot performance on rare categories. The rest types of methods utilize learning strategies for specific open vocabulary datasets and are collectively classified as Learning Strategy Based Method.
• Figure 5: The timeline of VLM-based segmentation methods.