Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem

TL;DR

This work analyzes why vision-language models struggle with basic multi-object reasoning by framing failures through the binding problem. Through four experiments—visual search, numerosity estimation, scene description, and visual analogy—the authors demonstrate that representational interference and binding errors limit rapid multi-object processing, mirroring human constraints. They introduce a scene-description benchmark and show that reducing binding interference via input structuring improves performance, while discussing the trade-offs of compositional representations and potential paths to improved reasoning. The findings imply that VLMs possess compositional representations but require mechanisms (e.g., serial processing or object-centric frameworks) to manage bindings without sacrificing generalization, guiding future model design and evaluation.

Abstract

Recent work has documented striking heterogeneity in the performance of state-of-the-art vision language models (VLMs), including both multimodal language models and text-to-image models. These models are able to describe and generate a diverse array of complex, naturalistic images, yet they exhibit surprising failures on basic multi-object reasoning tasks -- such as counting, localization, and simple forms of visual analogy -- that humans perform with near perfect accuracy. To better understand this puzzling pattern of successes and failures, we turn to theoretical accounts of the binding problem in cognitive science and neuroscience, a fundamental problem that arises when a shared set of representational resources must be used to represent distinct entities (e.g., to represent multiple objects in an image), necessitating the use of serial processing to avoid interference. We find that many of the puzzling failures of state-of-the-art VLMs can be explained as arising due to the binding problem, and that these failure modes are strikingly similar to the limitations exhibited by rapid, feedforward processing in the human brain.

Figures (9)

• Figure 1: Visual search tasks and results. Example trials for the 2D (top) and 3D (bottom) variants of the disjunctive (left/red column) and conjunctive (middle/blue column) search conditions. Performance for 2D and 3D task variants are plotted on the right. Results reflect aggregate performance for all four VLMs (GPT-4v, GPT-4o, Gemini Ultra 1.5, and Claude Sonnet 3.5; see Supplementary Figure \ref{['search_supp_figure1']} for separate model results). Error bars denote 95% binomial confidence intervals.
• Figure 2: Numerical estimation tasks and results. Top left: Examples of images generated by text-to-image (T2I) models for different numbers and categories of objects. Top right: Performance of T2I models as a function of the number and category of objects. Results reflect an aggregate of four models (Stable Diffusion Ultra, DALL-E 3, Google Parti, and Google Muse). Bottom left: Examples of images (featuring either 2D or 3D objects) used to evaluate numerosity estimation. Feature entropy was varied in four conditions (low entropy, high entropy, and two medium entropy conditions). Bottom middle: Numerosity estimation results for four multimodal language models (GPT-4v, GPT-4o, Gemini Ultra 1.5, Claude Sonnet 3.5; see Supplementary Figure \ref{['llava_supp_figure']} for results with Llava-1.5). Bottom right: Numerosity estimation results plotted as a function of the number of objects in an image, aggregated across all four models (see Supplementary Figure \ref{['numerosity_supp_figure']} for individual model results). Error bars for all plots reflect 95% binomial confidence intervals.
• Figure 3: Scene description task and results. A) Example image used in 2D scene description task, illustrating the concept of feature triplets: sets of three objects where one pair of objects shares a feature, and another pair shares a different feature. This example contains three feature triplets, demarcated by the dashed lines. 3D scenes were also investigated. B) Scene description results for text-to-image (T2I) models) as a function of the number of feature triplets. C) 2D scene description results for multimodal language models as a function of the number of feature triplets. Left panel illustrates the results aggregated across four models (GPT-4v, GPT-4o, Gemini Ultra 1.5, and Claude Sonnet 3.5). Right panel illustrates the results aggregated across scenes with different numbers of objects. D) 3D scene description results. Error bars represent the standard error of the mean.
• Figure 4: Visual analogy task. The Unified and Decomposed conditions present the same object pairs, but in the Decomposed condition it is broken up across three images. The correct target pair must share both relations (shape and color) with the source pair, so the correct answer in this example is Target Pair 2 because it satisfies both the ‚Äòsame shape‚Äô and ‚Äòdifferent color‚Äô relations.
• Figure 5: Visual search model results. Individual model performance for 2D and 3D visual search tasks. Error bars denote 95% binomial confidence intervals.