VISaGE: Understanding Visual Generics and Exceptions

TL;DR

The paper addresses how vision-language models reconcile category-level generics with instance-level visual cues when faced with atypical examples. It introduces VISaGE, a dataset of norm–exception text-image pairs that enables controlled probing of conceptual versus instance attribute reasoning under text-image congruence and incongruence across multiple open-weight VLMs, complemented by Shapley-based input attribution. Key findings show that pragmatic priors often overshadow conceptual knowledge in incongruent settings, while semantic priors influence instance recognition; Shapley analyses indicate models rely on visual cues but remain anchored to generic semantics. The work highlights fundamental limitations of current VLM grounding and provides a benchmark and analytic methods to guide future improvements for robust handling of within-category variation and generics in multimodal systems.

Abstract

While Vision Language Models (VLMs) learn conceptual representations, in the form of generalized knowledge, during training, they are typically used to analyze individual instances. When evaluation instances are atypical, this paradigm results in tension between two priors in the model. The first is a pragmatic prior that the textual and visual input are both relevant, arising from VLM finetuning on congruent inputs; the second is a semantic prior that the conceptual representation is generally true for instances of the category. In order to understand how VLMs trade off these priors, we introduce a new evaluation dataset, VISaGE, consisting of both typical and exceptional images. In carefully balanced experiments, we show that conceptual understanding degrades when the assumption of congruency underlying the pragmatic prior is violated with incongruent images. This effect is stronger than the effect of the semantic prior when querying about individual instances.

Figures (5)

• Figure 1: VISaGE contains both typical and exceptional instances of a category, with respect to some conceptual norm. We probe VLMs for conceptual and instance-level understanding, with congruent (top) and incongruent (middle, bottom) text-image pairs. Model failures indicate effects of both pragmatic and semantic biases.
• Figure 2: Summary of experiments and conditions: Exp. ① tests models' conceptual understanding of generics, while Exp. ② tests models' ability to reason about instances.
• Figure 3: Results: See Fig. \ref{['fig:experiments']} for setup. Exp 1: Conceptual attribute prediction accuracy decreases for incongruent inputs (b and d). Exp 2: Across most models, instance attribute recognition declines for exceptional images (b), unless they are named as such (c). Fig 3iii shows data from Exps 1&2: The difference between conceptual and instance accuracy is highest for incongruent pairs (b). Most models have higher instance accuracy in condition (b). Models are ordered by size. Numbers in Appendix \ref{['app:full-results']}.
• Figure 4: MM-SHAP calculated with smolVLM2. V-SHAP measures the proportion of Shapley values coming from the image, while T-SHAP is the proportion from text tokens.
• Figure 5: Annotation interface for dataset validation and expansion