Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision language models have difficulty recognizing virtual objects

Tyler Tran, Sangeet Khemlani, J. G. Trafton

TL;DR

This paper investigates whether vision-language models can perform imaginative reasoning about virtual objects not depicted in images. It introduces a TableTest-based benchmarking protocol with seven prompt variants to probe virtual object reasoning across three state-of-the-art VLMs. Results reveal substantial limitations, with peak accuracies around 63% for Idefics2, 57% for Llama3, and 22% for BLIP, and show that tense and numerical cues influence performance. The study argues that current VLMs lack robust visuospatial imagination and discusses hybrid symbolic-distributed architectures as a path toward mental-model-like scene understanding.

Abstract

Vision language models (VLMs) are AI systems paired with both language and vision encoders to process multimodal input. They are capable of performing complex semantic tasks such as automatic captioning, but it remains an open question about how well they comprehend the visuospatial properties of scenes depicted in the images they process. We argue that descriptions of virtual objects -- objects that are not visually represented in an image -- can help test scene comprehension in these AI systems. For example, an image that depicts a person standing under a tree can be paired with the following prompt: imagine that a kite is stuck in the tree. VLMs that comprehend the scene should update their representations and reason sensibly about the spatial relations between all three objects. We describe systematic evaluations of state-of-the-art VLMs and show that their ability to process virtual objects is inadequate.

Vision language models have difficulty recognizing virtual objects

TL;DR

This paper investigates whether vision-language models can perform imaginative reasoning about virtual objects not depicted in images. It introduces a TableTest-based benchmarking protocol with seven prompt variants to probe virtual object reasoning across three state-of-the-art VLMs. Results reveal substantial limitations, with peak accuracies around 63% for Idefics2, 57% for Llama3, and 22% for BLIP, and show that tense and numerical cues influence performance. The study argues that current VLMs lack robust visuospatial imagination and discusses hybrid symbolic-distributed architectures as a path toward mental-model-like scene understanding.

Abstract

Vision language models (VLMs) are AI systems paired with both language and vision encoders to process multimodal input. They are capable of performing complex semantic tasks such as automatic captioning, but it remains an open question about how well they comprehend the visuospatial properties of scenes depicted in the images they process. We argue that descriptions of virtual objects -- objects that are not visually represented in an image -- can help test scene comprehension in these AI systems. For example, an image that depicts a person standing under a tree can be paired with the following prompt: imagine that a kite is stuck in the tree. VLMs that comprehend the scene should update their representations and reason sensibly about the spatial relations between all three objects. We describe systematic evaluations of state-of-the-art VLMs and show that their ability to process virtual objects is inadequate.
Paper Structure (4 sections, 3 figures, 1 table)

This paper contains 4 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Benchmarking methodology for testing virtual object recognition
  3. Evaluation study results
  4. Discussion

Figures (3)

  • Figure 1: Examples from the TableTest dataset, which includes 64 individual objects (a) in various 2-object (b) and 3-object configurations (c).
  • Figure 2: Proportions of accuracy from evaluations of 7 separate prompts concerning virtual object recognition in 2-object images from TableTest. Lines in each panel depict overall accuracies, density plots depict performance distributions across TableTest's 64 objects, and bars depict histograms across those objects, as organized by whether the object served as the leftmost object in images. Humanlike performance estimated at ceiling (accuracy = 1.0).
  • Figure 3: Proportions of accuracy from evaluations of present vs. past versions of prompts (top panel) and for no cue vs. numerical cue formulations; density plots depict performance distributions across 64 objects, and bars show accuracy histograms for those objects, as organized by the leftmost object in images. Humanlike performance estimated at ceiling (accuracy = 1.0).