Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy

TL;DR

This work tackles the difficulty of evaluating open-ended text-generative vision-language models by identifying limitations of existing VQA benchmarks and proposing a granular, taxonomy-informed approach. It introduces an open-ended VQA benchmark that transforms classification datasets into VQA tasks, employs visual context cropping, and uses semantic-hierarchy follow-ups to resolve ambiguity in fine-grained predictions. The authors compare Text-VLMs with discriminative VLMs across object, action, and attribute tasks, evaluate multiple NLP/LLM-based metrics, and validate them against human judgments to select robust evaluation criteria. The framework yields a stratified, interpretable assessment of model capabilities and reveals complementary strengths between generic pre-trained models and VQA-specialized models, guiding targeted improvements in vision-language benchmarking and model development.

Abstract

The evaluation of text-generative vision-language models is a challenging yet crucial endeavor. By addressing the limitations of existing Visual Question Answering (VQA) benchmarks and proposing innovative evaluation methodologies, our research seeks to advance our understanding of these models' capabilities. We propose a novel VQA benchmark based on well-known visual classification datasets which allows a granular evaluation of text-generative vision-language models and their comparison with discriminative vision-language models. To improve the assessment of coarse answers on fine-grained classification tasks, we suggest using the semantic hierarchy of the label space to ask automatically generated follow-up questions about the ground-truth category. Finally, we compare traditional NLP and LLM-based metrics for the problem of evaluating model predictions given ground-truth answers. We perform a human evaluation study upon which we base our decision on the final metric. We apply our benchmark to a suite of vision-language models and show a detailed comparison of their abilities on object, action, and attribute classification. Our contributions aim to lay the foundation for more precise and meaningful assessments, facilitating targeted progress in the exciting field of vision-language modeling.

Figures (14)

• Figure 1: Our proposed framework for building an open-ended VQA benchmark using visual guidance for question context and probing for details with follow-up questions.
• Figure 2: Comparison of Models on open-ended VQA datasets, testing traditional VQA, Object-oVQA, Activity-oVQA, and Attribute-oVQA. Generic pretrained models like BLIP-2 are stronger on predicting objects and activities with high semantic granularity while exhibiting lower performances for Attribute-oVQA and classical VQA tasks. Contrary, VQA pretrained models are stronger on predicting attribute concepts and answering generic VQA questions but perform poorly for fine-granularity of nouns and activities.
• Figure 3: OVAD-oVQA questions. For every attribute type we build three different questions to evaluate the Text-VLMs. We show an example of the three different questions for every type of attribute. The word object is replaced by the noun category for every question.
• Figure 4: Qualitative results for ImageNet-oVQA. Only the bounding box crop is considered as input image for the model prediction. Coloring: Answers are considered correct / wrong under ClipM metric.
• Figure 5: Qualitative examples for the Activity oVQA task. Figures (a) & (b) show how the follow-up question makes use of the hierarchical parent to find the exact answer. Figures (c) & (d) show the outputs of different VLM models for the Activity-oVQA task. Coloring: Answers are considered correct / wrong under ClipM metric.