CLIP-UP: CLIP-Based Unanswerable Problem Detection for Visual Question Answering

TL;DR

This work tackles the unreliability of vision-language models in answering unanswerable visual questions by introducing CLIP-UP, a lightweight method that leverages Structure-CLIP embeddings to create correlation signals between the image and the question (and options). A learnable, minimal projection layer (and optionally Injected LoRA) converts these signals into an answerability prior embedded into the VLM, enabling the model to withhold answers when appropriate without changing existing weights. CLIP-UP achieves significant improvements on Unsolvable Problem Detection across both multiple-choice and open-ended VQA benchmarks, across several model families, while preserving performance on standard tasks. It also introduces a dataset for training UPD in the multiple-choice setting and demonstrates an inference-time control to trade off standard and dual accuracies. Overall, CLIP-UP offers a data-efficient, generalizable approach to robust VQA evaluation and reliable downstream deployment of VLMs.

Abstract

Vision-Language Models (VLMs) demonstrate remarkable capabilities in visual understanding and reasoning, such as in Visual Question Answering (VQA), where the model is asked a question related to a visual input. Still, these models can make distinctly unnatural errors, for example, providing (wrong) answers to unanswerable VQA questions, such as questions asking about objects that do not appear in the image. To address this issue, we propose CLIP-UP: CLIP-based Unanswerable Problem detection, a novel lightweight method for equipping VLMs with the ability to withhold answers to unanswerable questions. CLIP-UP leverages CLIP-based similarity measures to extract question-image alignment information to detect unanswerability, requiring efficient training of only a few additional layers, while keeping the original VLMs' weights unchanged. Tested across several models, CLIP-UP achieves significant improvements on benchmarks assessing unanswerability in both multiple-choice and open-ended VQA, surpassing other methods, while preserving original performance on other tasks.

Figures (10)

• Figure 1: CLIP-UP equips VLMs such as LLaVA-1.5-7B liu2023improvedllava with the ability to detect and withhold answers to multiple-choice and open-ended unanswerable questions, while preserving models' original capabilities on standard answerable questions.
• Figure 2: CLIP-UP embedding injection applied on common VLM architectures. (a) For multiple-choice questions, given an image and a VQA prompt, the prompt is transformed into text segments merging the question with each answer option. These segments and the image are encoded by Structure-CLIP (S-CLIP) to produce embeddings, from which correlation vectors are formed via element-wise multiplication. A learnable projection module maps these vectors into the VLM's intermediate feature space. The resulting new embedding vector is integrated into the LLM component of the VLM alongside the standard inputs. (b) For open-ended questions, the process is similar but involves a single correlation vector computed from the image and question.
• Figure 3: Our correlation vectors capture a prior for VQA answerability. In multiple-choice questions, (a) for a standard question, the correct contextualized answer option aligns well with the image, resulting in a correlation vector with high values. For unanswerable questions, no option aligns well with the image: either (b) all answer options are incorrect (AAD), (c) incorrect and irrelevant to the question (IASD), or (d) the question is incompatible with the image (IVQD). Open-ended questions show a similar trend: (e) answerable questions align well with the image, in contrast to (f) unanswerable ones. Numbers represent average Structure-CLIP similarity scores measured on test data.
• Figure 4: t-SNE plots of embedding vectors generated by CLIP-UP on LLaVA-1.5-7B, for (a-c) all samples in MM-UPD, and (d) all samples in the RGQA test data.
• Figure 5: Pairs of standard and unanswerable multiple-choice VQA questions from our multiple-choice dataset for (a) AAD, (b) IASD, and (c) IVQD.