Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hidden Meanings in Plain Sight: RebusBench for Evaluating Cognitive Visual Reasoning

Seyed Amir Kasaei, Arash Marioriyad, Mahbod Khaleti, MohammadAmin Fazli, Mahdieh Soleymani Baghshah, Mohammad Hossein Rohban

Abstract

Large Vision-Language Models (LVLMs) have achieved remarkable proficiency in explicit visual recognition, effectively describing what is directly visible in an image. However, a critical cognitive gap emerges when the visual input serves only as a clue rather than the answer. We identify that current models struggle with the complex, multi-step reasoning required to solve problems where information is not explicitly depicted. Successfully solving a rebus puzzle requires a distinct cognitive workflow: the model must extract visual and textual attributes, retrieve linguistic prior knowledge (such as idioms), and perform abstract mapping to synthesize these elements into a meaning that exists outside the pixel space. To evaluate this neurosymbolic capability, we introduce RebusBench, a benchmark of 1,164 puzzles designed to test this specific integration of perception and knowledge. Our evaluation of state-of-the-art models (including Qwen, InternVL, and LLaVA) shows a severe deficiency: performance saturates below 10% Exact Match and 20% semantic accuracy, with no significant improvement observed from model scaling or In-Context Learning (ICL). These findings suggest that while models possess the necessary visual and linguistic components, they lack the cognitive reasoning glue to connect them. Project page available at https://amirkasaei.com/rebusbench/.

Hidden Meanings in Plain Sight: RebusBench for Evaluating Cognitive Visual Reasoning

Abstract

Large Vision-Language Models (LVLMs) have achieved remarkable proficiency in explicit visual recognition, effectively describing what is directly visible in an image. However, a critical cognitive gap emerges when the visual input serves only as a clue rather than the answer. We identify that current models struggle with the complex, multi-step reasoning required to solve problems where information is not explicitly depicted. Successfully solving a rebus puzzle requires a distinct cognitive workflow: the model must extract visual and textual attributes, retrieve linguistic prior knowledge (such as idioms), and perform abstract mapping to synthesize these elements into a meaning that exists outside the pixel space. To evaluate this neurosymbolic capability, we introduce RebusBench, a benchmark of 1,164 puzzles designed to test this specific integration of perception and knowledge. Our evaluation of state-of-the-art models (including Qwen, InternVL, and LLaVA) shows a severe deficiency: performance saturates below 10% Exact Match and 20% semantic accuracy, with no significant improvement observed from model scaling or In-Context Learning (ICL). These findings suggest that while models possess the necessary visual and linguistic components, they lack the cognitive reasoning glue to connect them. Project page available at https://amirkasaei.com/rebusbench/.

Paper Structure

This paper contains 21 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Scene Understanding and Image Captioning.
  4. Spatial and Compositional Reasoning.
  5. Visual Counting and Quantity Estimation.
  6. Advanced Reasoning and Multi-discipline Benchmarks.
  7. The RebusBench Dataset
  8. Experiments
  9. Experimental Setup
  10. Models.
  11. Prompting Strategies.
  12. Evaluation Metrics.
  13. Results
  14. Overall Performance and Model Scaling.
  15. Impact of In-Context Learning.
  16. ...and 6 more sections

Figures (1)

  • Figure 1: Sample puzzles from RebusBench. Solving these requires transcending literal perception ("System 1") to perform abstract "System 2" reasoning, where models must entangle visual cues (position, color, style) with linguistic knowledge to reconstruct the hidden idiom.