Sensitivity of Generative VLMs to Semantically and Lexically Altered Prompts

TL;DR

It is demonstrated that generative VLMs are highly sensitive to lexical alterations in prompts without corresponding semantic changes, and this vulnerability affects the performance of techniques aimed at achieving consistency in their outputs.

Abstract

Despite the significant influx of prompt-tuning techniques for generative vision-language models (VLMs), it remains unclear how sensitive these models are to lexical and semantic alterations in prompts. In this paper, we evaluate the ability of generative VLMs to understand lexical and semantic changes in text using the SugarCrepe++ dataset. We analyze the sensitivity of VLMs to lexical alterations in prompts without corresponding semantic changes. Our findings demonstrate that generative VLMs are highly sensitive to such alterations. Additionally, we show that this vulnerability affects the performance of techniques aimed at achieving consistency in their outputs.

Figures (1)

• Figure 1: Examples from SugarCrepe++ (SC++) dataset. $P_1$ and $P_2$ are semantically equivalent but lexically different while $N$ is semantically different than both $P_1$ and $P_2$ despite its lexical similarity with $P_1$.