Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond General Prompts: Automated Prompt Refinement using Contrastive Class Alignment Scores for Disambiguating Objects in Vision-Language Models

Lucas Choi, Ross Greer

TL;DR

Problem: Prompt sensitivity in zero-shot vision-language models leads to inconsistent object detection performance. Approach: Introduces Contrastive Class Alignment Scores (CCAS) that compare LLM-generated prompts against the target class vs confounds using embeddings from a sentence transformer, with two variants $CCAS_{avg}$ and $CCAS_{max}$ defined as $CCAS_{avg}(t_i) = \cos(\vec{t_i}, \vec{T}) - \frac{1}{NM} \sum_{m=1}^{M} \sum_{k=1}^{N} \cos(\vec{t_i}, \vec{c}_{m,k})$ and $CCAS_{max}(t_i) = \cos(\vec{t_i}, \vec{T}) - \max_{m,k} \cos(\vec{t_i}, \vec{c}_{m,k})$. They avoid model fine-tuning and rely on zero-shot inference with a model-agnostic pipeline. Contributions: A CCAS-based prompt-filtering pipeline using LLM-generated candidates and sentence-embedding comparisons, demonstrating improved average precision on two datasets; interpretable by tying performance to semantic distances between target and confounds. Impact: scalable, model-agnostic, and reduces manual prompt engineering for robust VLM-based detection tasks.

Abstract

Vision-language models (VLMs) offer flexible object detection through natural language prompts but suffer from performance variability depending on prompt phrasing. In this paper, we introduce a method for automated prompt refinement using a novel metric called the Contrastive Class Alignment Score (CCAS), which ranks prompts based on their semantic alignment with a target object class while penalizing similarity to confounding classes. Our method generates diverse prompt candidates via a large language model and filters them through CCAS, computed using prompt embeddings from a sentence transformer. We evaluate our approach on challenging object categories, demonstrating that our automatic selection of high-precision prompts improves object detection accuracy without the need for additional model training or labeled data. This scalable and model-agnostic pipeline offers a principled alternative to manual prompt engineering for VLM-based detection systems.

Beyond General Prompts: Automated Prompt Refinement using Contrastive Class Alignment Scores for Disambiguating Objects in Vision-Language Models

TL;DR

Problem: Prompt sensitivity in zero-shot vision-language models leads to inconsistent object detection performance. Approach: Introduces Contrastive Class Alignment Scores (CCAS) that compare LLM-generated prompts against the target class vs confounds using embeddings from a sentence transformer, with two variants and defined as and . They avoid model fine-tuning and rely on zero-shot inference with a model-agnostic pipeline. Contributions: A CCAS-based prompt-filtering pipeline using LLM-generated candidates and sentence-embedding comparisons, demonstrating improved average precision on two datasets; interpretable by tying performance to semantic distances between target and confounds. Impact: scalable, model-agnostic, and reduces manual prompt engineering for robust VLM-based detection tasks.

Abstract

Vision-language models (VLMs) offer flexible object detection through natural language prompts but suffer from performance variability depending on prompt phrasing. In this paper, we introduce a method for automated prompt refinement using a novel metric called the Contrastive Class Alignment Score (CCAS), which ranks prompts based on their semantic alignment with a target object class while penalizing similarity to confounding classes. Our method generates diverse prompt candidates via a large language model and filters them through CCAS, computed using prompt embeddings from a sentence transformer. We evaluate our approach on challenging object categories, demonstrating that our automatic selection of high-precision prompts improves object detection accuracy without the need for additional model training or labeled data. This scalable and model-agnostic pipeline offers a principled alternative to manual prompt engineering for VLM-based detection systems.
Paper Structure (6 sections, 2 equations, 3 figures, 4 tables)

This paper contains 6 sections, 2 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Research
  3. Algorithm
  4. Experimental Evaluation
  5. Discussion
  6. Concluding Remarks

Figures (3)

  • Figure 1: This is a sample detection from foundation VLM OWLv2 prompted with 'goggles'. The model mistakenly detected these sunglasses as goggles, which may have serious safety implications in a worksite monitoring task where safety goggles are important. As illustrated, certain prompts have ambiguity in their definitions, reflecting the many-object-encompassing aspect of natural language, but also resulting in poor precision detections and necessitating more descriptive prompts to ensure unintended objects are not mistakenly detected.
  • Figure 2: System Diagram of our proposed algorithmic pipeline for identifying an optimal high-precision prompt through our CCAS metric. The diagram utilizes goggles as the example classes and prompts, with only a sample of prompts being shown in the diagram. The complete pipeline and illustrated example are discussed in Algorithm and Experimental Evaluation.
  • Figure 3: Similarity Matrix of the Goggle Detection task prompts, specifically between the 'goggle', 'glasses', and 'sunglasses' classes. The y-axis consists of the target class prompts, while the x-axis consists of all of the prompts from the confounding classes.