Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Training-Free Guess What Vision Language Model from Snippets to Open-Vocabulary Object Detection

Guiying Zhu, Bowen Yang, Yin Zhuang, Tong Zhang, Guanqun Wang, Zhihao Che, He Chen, Lianlin Li

TL;DR

This work tackles open-vocabulary object detection without additional training by leveraging pre-trained foundation models. It introduces GW-VLM, which couples a Vision-Language Model with a Large Language Model through a Multi-Scale Visual Language Searching module and a Contextual Concept Prompt to build a universal object understanding at inference. The method combines Class-Agnostic RPNs, MS-VLS with soft visual-language alignment, and CCP to enable an inference-time 'guess what' interaction between VLM and LLM, yielding state-of-the-art results on remote sensing datasets and competitive performance on natural-scene benchmarks while avoiding training on task-specific data. The approach demonstrates strong cross-domain generalization and reduces reliance on costly pre-training and labeling, offering a practical path toward truly open-vocabulary detection.

Abstract

Open-Vocabulary Object Detection (OVOD) aims to develop the capability to detect anything. Although myriads of large-scale pre-training efforts have built versatile foundation models that exhibit impressive zero-shot capabilities to facilitate OVOD, the necessity of creating a universal understanding for any object cognition according to already pretrained foundation models is usually overlooked. Therefore, in this paper, a training-free Guess What Vision Language Model, called GW-VLM, is proposed to form a universal understanding paradigm based on our carefully designed Multi-Scale Visual Language Searching (MS-VLS) coupled with Contextual Concept Prompt (CCP) for OVOD. This approach can engage a pre-trained Vision Language Model (VLM) and a Large Language Model (LLM) in the game of "guess what". Wherein, MS-VLS leverages multi-scale visual-language soft-alignment for VLM to generate snippets from the results of class-agnostic object detection, while CCP can form the concept of flow referring to MS-VLS and then make LLM understand snippets for OVOD. Finally, the extensive experiments are carried out on natural and remote sensing datasets, including COCO val, Pascal VOC, DIOR, and NWPU-10, and the results indicate that our proposed GW-VLM can achieve superior OVOD performance compared to the-state-of-the-art methods without any training step.

A Training-Free Guess What Vision Language Model from Snippets to Open-Vocabulary Object Detection

TL;DR

This work tackles open-vocabulary object detection without additional training by leveraging pre-trained foundation models. It introduces GW-VLM, which couples a Vision-Language Model with a Large Language Model through a Multi-Scale Visual Language Searching module and a Contextual Concept Prompt to build a universal object understanding at inference. The method combines Class-Agnostic RPNs, MS-VLS with soft visual-language alignment, and CCP to enable an inference-time 'guess what' interaction between VLM and LLM, yielding state-of-the-art results on remote sensing datasets and competitive performance on natural-scene benchmarks while avoiding training on task-specific data. The approach demonstrates strong cross-domain generalization and reduces reliance on costly pre-training and labeling, offering a practical path toward truly open-vocabulary detection.

Abstract

Open-Vocabulary Object Detection (OVOD) aims to develop the capability to detect anything. Although myriads of large-scale pre-training efforts have built versatile foundation models that exhibit impressive zero-shot capabilities to facilitate OVOD, the necessity of creating a universal understanding for any object cognition according to already pretrained foundation models is usually overlooked. Therefore, in this paper, a training-free Guess What Vision Language Model, called GW-VLM, is proposed to form a universal understanding paradigm based on our carefully designed Multi-Scale Visual Language Searching (MS-VLS) coupled with Contextual Concept Prompt (CCP) for OVOD. This approach can engage a pre-trained Vision Language Model (VLM) and a Large Language Model (LLM) in the game of "guess what". Wherein, MS-VLS leverages multi-scale visual-language soft-alignment for VLM to generate snippets from the results of class-agnostic object detection, while CCP can form the concept of flow referring to MS-VLS and then make LLM understand snippets for OVOD. Finally, the extensive experiments are carried out on natural and remote sensing datasets, including COCO val, Pascal VOC, DIOR, and NWPU-10, and the results indicate that our proposed GW-VLM can achieve superior OVOD performance compared to the-state-of-the-art methods without any training step.
Paper Structure (12 sections, 7 equations, 15 figures, 8 tables)

This paper contains 12 sections, 7 equations, 15 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Class-Agnostic RPNs
  5. Multi-Scale Visual Language Searching
  6. Contextual Concept Prompt
  7. Experiments
  8. Implementation Details
  9. Comparison Results
  10. Ablation Study
  11. Open Vocabulary Analysis
  12. Conclusion

Figures (15)

  • Figure 1: The pipeline of our ‘Guess What’ game: To understand class-agnostic object based on snippets for OVOD.
  • Figure 2: Overview of GW-VLM. (A) By merging Class-Agnostic RPNs, capturing class-agnostic objects from the input images. (B) Based on the Multi-Scale Visual Language Searching, the extracted crops are soft-aligned with the text tokens in VLM to generate snippets by selecting Top-K semantically matched phrases. (C) The Top-K matched snippets are subsequently embedded into the Contextual Concept Prompt (CCP), which incorporates background information and visual searching information. (D) LLM conducts semantic reasoning and prediction using the CCP.
  • Figure 3: Visualization of natural and remote sensing objects detection results. Ours refers to the proposed GW-VLM. This figure presents a qualitative comparison of various models on the DIOR and COCO datasets.
  • Figure 4: P-R curves of detector fusion on four datasets: (a) DIOR, (b) NWPU-10, (c) COCO, (d) Pascal VOC.
  • Figure 5: Prompt templates for both remote sensing and natural scenes
  • ...and 10 more figures