Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Far Are We from Intelligent Visual Deductive Reasoning?

Yizhe Zhang, He Bai, Ruixiang Zhang, Jiatao Gu, Shuangfei Zhai, Josh Susskind, Navdeep Jaitly

TL;DR

This study benchmarks Vision-Language Models on Raven’s Progressive Matrices to probe visual deductive reasoning beyond standard VQA tasks. It reveals a persistent gap to human performance, with perception identified as the main bottleneck; even strong models struggle with multi-hop relational reasoning on abstract RPM patterns. The authors demonstrate that segmentation of RPMs, oracle text descriptions, and carefully structured prompts can improve reasoning, but do not bridge the gap completely. The work provides a systematic methodology and datasets for future improvements in visual deduction and hypothesis verification for VLMs.

Abstract

Vision-Language Models (VLMs) have recently demonstrated incredible strides on diverse vision language tasks. We dig into vision-based deductive reasoning, a more sophisticated but less explored realm, and find previously unexposed blindspots in the current SOTA VLMs. Specifically, we leverage Raven's Progressive Matrices (RPMs), to assess VLMs' abilities to perform multi-hop relational and deductive reasoning relying solely on visual clues. We perform comprehensive evaluations of several popular VLMs employing standard strategies such as in-context learning, self-consistency, and Chain-of-thoughts (CoT) on three diverse datasets, including the Mensa IQ test, IntelligenceTest, and RAVEN. The results reveal that despite the impressive capabilities of LLMs in text-based reasoning, we are still far from achieving comparable proficiency in visual deductive reasoning. We found that certain standard strategies that are effective when applied to LLMs do not seamlessly translate to the challenges presented by visual reasoning tasks. A detailed analysis reveals that VLMs struggle to solve these tasks mainly because they are unable to perceive and comprehend multiple, confounding abstract patterns in RPM examples.

How Far Are We from Intelligent Visual Deductive Reasoning?

TL;DR

This study benchmarks Vision-Language Models on Raven’s Progressive Matrices to probe visual deductive reasoning beyond standard VQA tasks. It reveals a persistent gap to human performance, with perception identified as the main bottleneck; even strong models struggle with multi-hop relational reasoning on abstract RPM patterns. The authors demonstrate that segmentation of RPMs, oracle text descriptions, and carefully structured prompts can improve reasoning, but do not bridge the gap completely. The work provides a systematic methodology and datasets for future improvements in visual deduction and hypothesis verification for VLMs.

Abstract

Vision-Language Models (VLMs) have recently demonstrated incredible strides on diverse vision language tasks. We dig into vision-based deductive reasoning, a more sophisticated but less explored realm, and find previously unexposed blindspots in the current SOTA VLMs. Specifically, we leverage Raven's Progressive Matrices (RPMs), to assess VLMs' abilities to perform multi-hop relational and deductive reasoning relying solely on visual clues. We perform comprehensive evaluations of several popular VLMs employing standard strategies such as in-context learning, self-consistency, and Chain-of-thoughts (CoT) on three diverse datasets, including the Mensa IQ test, IntelligenceTest, and RAVEN. The results reveal that despite the impressive capabilities of LLMs in text-based reasoning, we are still far from achieving comparable proficiency in visual deductive reasoning. We found that certain standard strategies that are effective when applied to LLMs do not seamlessly translate to the challenges presented by visual reasoning tasks. A detailed analysis reveals that VLMs struggle to solve these tasks mainly because they are unable to perceive and comprehend multiple, confounding abstract patterns in RPM examples.
Paper Structure (24 sections, 4 figures, 6 tables)

This paper contains 24 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Experiment Setting
  4. Evaluation Results
  5. Evaluation of VLMs on Visual Deductive Reasoning
  6. Uncertainty of the prediction
  7. Do standard strategies in LLMs translate effectively to visual deductive reasoning?
  8. VLMs struggle with reading in-context image
  9. What limits the performance of the VLMs?
  10. How good is the VLM's perception on this task?
  11. How good is the VLM's deductive reasoning on this task?
  12. Does the oracle text description improve the model's performance?
  13. Will removing the visual cues harm the model?
  14. Can the performance be improved by reasoning with noisy text descriptions generated by the model itself?
  15. How good is the VLM's hypothesis verification on this task?
  16. ...and 9 more sections

Figures (4)

  • Figure 1: Illustration of the visual deductive reasoning for Raven’s Progressive Matrices. The task requires intricate coordination among perception, deductive reasoning, and hypothesis verification capabilities exhibited by Vision-Language Models.
  • Figure 2: Three manually created RPM problems evaluated for text description augmentation, illustrating varying levels of difficulty. The correct answers are "F, F, F".
  • Figure 3: Accuracy of the original RPM as input with that of the segmented RPM as input. Results based on 10 repetitions.
  • Figure 4: K-shot evaluation results with Raven subset.