Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Cognitive Paradigm Approach to Probe the Perception-Reasoning Interface in VLMs

Mohit Vaishnav, Tanel Tammet

TL;DR

This work proposes a cognitively inspired framework to diagnose the perception–reasoning interface in Vision-Language Models by employing three paradigms—Direct Visual Rule Learning, Deductive Rule Learning, and Componential Analysis. CA uniquely decouples perception from reasoning by converting images to rich, task-agnostic textual descriptions and then applying reasoning with powerful LLMs, yielding SOTA performance on Bongard-OpenWorld, Bongard-HOI, and Winoground. Ablations reveal a pronounced perception bottleneck in open-source VLMs, with substantial gains when perception is replaced by high-quality descriptions or when reasoning is performed with external descriptions or text-only LLMs. The results advocate modular architectures that separate perception from symbolic reasoning, highlighting a path toward more robust and general visual intelligence in AI.

Abstract

A fundamental challenge in artificial intelligence involves understanding the cognitive mechanisms underlying visual reasoning in sophisticated models like Vision-Language Models (VLMs). How do these models integrate visual perception with abstract thought, especially when reasoning across multiple images or requiring fine-grained compositional understanding? Drawing inspiration from cognitive science, this paper introduces a structured evaluation framework using diverse visual reasoning tasks-Bongard Problems (BPs) and Winoground-to dissect the perception-reasoning interface in VLMs. We propose three distinct evaluation paradigms, mirroring human problem-solving strategies: Direct Visual Rule Learning (DVRL; holistic processing), Deductive Rule Learning (DRL; rule extraction and application), and Componential Analysis (CA; analytical decomposition via task-agnostic textual descriptions). These paradigms systematically vary cognitive load and probe processing stages. Notably, CA enables multi-image reasoning evaluation even for single-image architectures and isolates reasoning from perception by operating on textual descriptions. Applying this framework, we demonstrate that CA, leveraging powerful language models for reasoning over rich, independently generated descriptions, achieves new state-of-the-art (SOTA) performance on challenging benchmarks including Bongard-OpenWorld, Bongard-HOI, and Winoground. Ablation studies confirm reasoning improves significantly when perceptual challenges are mitigated, revealing a critical perception bottleneck. Our framework provides a valuable diagnostic tool and suggests that decoupling perception (via rich, task-agnostic description) from reasoning is a promising direction for robust and general visual intelligence.

A Cognitive Paradigm Approach to Probe the Perception-Reasoning Interface in VLMs

TL;DR

This work proposes a cognitively inspired framework to diagnose the perception–reasoning interface in Vision-Language Models by employing three paradigms—Direct Visual Rule Learning, Deductive Rule Learning, and Componential Analysis. CA uniquely decouples perception from reasoning by converting images to rich, task-agnostic textual descriptions and then applying reasoning with powerful LLMs, yielding SOTA performance on Bongard-OpenWorld, Bongard-HOI, and Winoground. Ablations reveal a pronounced perception bottleneck in open-source VLMs, with substantial gains when perception is replaced by high-quality descriptions or when reasoning is performed with external descriptions or text-only LLMs. The results advocate modular architectures that separate perception from symbolic reasoning, highlighting a path toward more robust and general visual intelligence in AI.

Abstract

A fundamental challenge in artificial intelligence involves understanding the cognitive mechanisms underlying visual reasoning in sophisticated models like Vision-Language Models (VLMs). How do these models integrate visual perception with abstract thought, especially when reasoning across multiple images or requiring fine-grained compositional understanding? Drawing inspiration from cognitive science, this paper introduces a structured evaluation framework using diverse visual reasoning tasks-Bongard Problems (BPs) and Winoground-to dissect the perception-reasoning interface in VLMs. We propose three distinct evaluation paradigms, mirroring human problem-solving strategies: Direct Visual Rule Learning (DVRL; holistic processing), Deductive Rule Learning (DRL; rule extraction and application), and Componential Analysis (CA; analytical decomposition via task-agnostic textual descriptions). These paradigms systematically vary cognitive load and probe processing stages. Notably, CA enables multi-image reasoning evaluation even for single-image architectures and isolates reasoning from perception by operating on textual descriptions. Applying this framework, we demonstrate that CA, leveraging powerful language models for reasoning over rich, independently generated descriptions, achieves new state-of-the-art (SOTA) performance on challenging benchmarks including Bongard-OpenWorld, Bongard-HOI, and Winoground. Ablation studies confirm reasoning improves significantly when perceptual challenges are mitigated, revealing a critical perception bottleneck. Our framework provides a valuable diagnostic tool and suggests that decoupling perception (via rich, task-agnostic description) from reasoning is a promising direction for robust and general visual intelligence.
Paper Structure (50 sections, 3 equations, 2 figures, 12 tables)

This paper contains 50 sections, 3 equations, 2 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Models
  4. Dataset and Task
  5. Cognitively-Inspired Evaluation Paradigms
  6. Direct Visual Rule Learning (DVRL)
  7. Deductive Rule Learning (DRL)
  8. Componential Analysis (CA)
  9. Results and Analysis
  10. Performance on Bongard-OpenWorld
  11. Performance on Bongard HOI
  12. Performance on Winoground
  13. Ablation Studies: Isolating Perception and Reasoning
  14. Rule Application Fidelity
  15. Impact of Description Quality on Reasoning
  16. ...and 35 more sections

Figures (2)

  • Figure 1: Example Bongard-OpenWorld task. Left: Positive examples. Center: Negative examples. Right: Query. Rule: A group photo at a wedding reception. Query is negative. (3 of 6 examples shown per set).
  • Figure 2: Cognitively-Inspired Evaluation Paradigms.DVRL (Direct Visual Rule Learning): Concurrent processing of all images, mimicking holistic perception. Requires multi-image input capability. DRL (Deductive Rule Learning): Two-stage process separating rule extraction from application, mimicking explicit deduction. CA (Componential Analysis): Multi-stage process involving individual image description followed by reasoning over text, mimicking analytical decomposition and enabling perception-reasoning separation.