WinoViz: Probing Visual Properties of Objects Under Different States

TL;DR

WinoViz presents a text-only benchmark to evaluate language models on object visual properties across different states, highlighting pragmatic and visual knowledge reasoning. The dataset (1,380 examples; 200 multi-hop variants) challenges models with single-hop and multi-hop scenarios, and the study compares zero-/few-shot performance across language and vision-language models, excluding image inputs. Findings show large language models excel in pragmatic reasoning but struggle with multi-hop visual knowledge reasoning, while vision-language models generally outperform text-only models; image-generation approaches provide limited utility. The work identifies visual-knowledge reasoning as a bottleneck and suggests that future progress will hinge on effectively transferring visual grounding into language models for more robust commonsense reasoning.

Abstract

Humans perceive and comprehend different visual properties of an object based on specific contexts. For instance, we know that a banana turns brown ``when it becomes rotten,'' whereas it appears green ``when it is unripe.'' Previous studies on probing visual commonsense knowledge have primarily focused on examining language models' understanding of typical properties (e.g., colors and shapes) of objects. We present WinoViz, a text-only evaluation dataset, consisting of 1,380 examples that probe the reasoning abilities of language models regarding variant visual properties of objects under different contexts or states. Our task is challenging since it requires pragmatic reasoning (finding intended meanings) and visual knowledge reasoning. We also present multi-hop data, a more challenging version of our data, which requires multi-step reasoning chains to solve our task. In our experimental analysis, our findings are: a) Large language models such as GPT-4 demonstrate effective performance, but when it comes to multi-hop data, their performance is significantly degraded. b) Large models perform well on pragmatic reasoning, but visual knowledge reasoning is a bottleneck in our task. c) Vision-language models outperform their language-model counterparts. d) A model with machine-generated images performs poorly in our task. This is due to the poor quality of the generated images.

Figures (8)

• Figure 1: The WinoViz task. We investigate the divergent properties of an object and explore the reasoning abilities of language models pertaining to object attributes. The premise sentence depicts a scene involving a banana and two hypothesis sentences describe the visual properties of a banana. The task is to choose a more plausible hypothesis given the premise. For the multi-hop version, we replace the visual attribute word with another object word which has a similar visual attribute.
• Figure 2: Dataset Collection. We collect our data through crowdsourcing efforts. The first step is to identify properties and visual attributes for an object and the second step is to write natural sentences for each property and attribute. Sentences with properties will be used as premise sentences and sentences with visual attributes will be used as hypothesis sentences.
• Figure 3: Examples of generated images. We generate images using Stable Diffusion rombach2022high. In the second example, the bananas in both images are yellow, leading the model to select the incorrect option. The generated image examples don't assist in selecting a more plausible hypothesis option.
• Figure 4: The Interface of the qualification task. We provide 12 questions to find quality workers.
• Figure 5: Interfaces of annotating visual contrast sets (parts 1 and 2).