Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reasoning Riddles: How Explainability Reveals Cognitive Limits in Vision-Language Models

Prahitha Movva

TL;DR

This work targets the opacity of reasoning in vision-language models when solving rebus puzzles. It introduces a 221-puzzle dataset annotated across six cognitive categories and six themes, plus an explainability-focused evaluation framework that decouples reasoning quality from answer accuracy. Through three prompting strategies, the study reveals pronounced cognitive bottlenecks—models excel at visual composition but struggle with absence reasoning and cultural symbolism—and shows that explicit cognitive scaffolding can improve reasoning traces and performance. The findings motivate interpretability-driven benchmarks and modular or neuro-symbolic architectures to enhance multimodal reasoning and trustworthy AI behavior.

Abstract

Vision-Language Models (VLMs) excel at many multimodal tasks, yet their cognitive processes remain opaque on complex lateral thinking challenges like rebus puzzles. While recent work has demonstrated these models struggle significantly with rebus puzzle solving, the underlying reasoning processes and failure patterns remain largely unexplored. We address this gap through a comprehensive explainability analysis that moves beyond performance metrics to understand how VLMs approach these complex lateral thinking challenges. Our study contributes a systematically annotated dataset of 221 rebus puzzles across six cognitive categories, paired with an evaluation framework that separates reasoning quality from answer correctness. We investigate three prompting strategies designed to elicit different types of explanatory processes and reveal critical insights into VLM cognitive processes. Our findings demonstrate that reasoning quality varies dramatically across puzzle categories, with models showing systematic strengths in visual composition while exhibiting fundamental limitations in absence interpretation and cultural symbolism. We also discover that prompting strategy substantially influences both cognitive approach and problem-solving effectiveness, establishing explainability as an integral component of model performance rather than a post-hoc consideration.

Reasoning Riddles: How Explainability Reveals Cognitive Limits in Vision-Language Models

TL;DR

This work targets the opacity of reasoning in vision-language models when solving rebus puzzles. It introduces a 221-puzzle dataset annotated across six cognitive categories and six themes, plus an explainability-focused evaluation framework that decouples reasoning quality from answer accuracy. Through three prompting strategies, the study reveals pronounced cognitive bottlenecks—models excel at visual composition but struggle with absence reasoning and cultural symbolism—and shows that explicit cognitive scaffolding can improve reasoning traces and performance. The findings motivate interpretability-driven benchmarks and modular or neuro-symbolic architectures to enhance multimodal reasoning and trustworthy AI behavior.

Abstract

Vision-Language Models (VLMs) excel at many multimodal tasks, yet their cognitive processes remain opaque on complex lateral thinking challenges like rebus puzzles. While recent work has demonstrated these models struggle significantly with rebus puzzle solving, the underlying reasoning processes and failure patterns remain largely unexplored. We address this gap through a comprehensive explainability analysis that moves beyond performance metrics to understand how VLMs approach these complex lateral thinking challenges. Our study contributes a systematically annotated dataset of 221 rebus puzzles across six cognitive categories, paired with an evaluation framework that separates reasoning quality from answer correctness. We investigate three prompting strategies designed to elicit different types of explanatory processes and reveal critical insights into VLM cognitive processes. Our findings demonstrate that reasoning quality varies dramatically across puzzle categories, with models showing systematic strengths in visual composition while exhibiting fundamental limitations in absence interpretation and cultural symbolism. We also discover that prompting strategy substantially influences both cognitive approach and problem-solving effectiveness, establishing explainability as an integral component of model performance rather than a post-hoc consideration.

Paper Structure

This paper contains 23 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Dataset collection and annotation
  4. Methodology
  5. Prompting strategies
  6. Evaluation framework
  7. Results
  8. Category-wise analysis
  9. Cognitive complexity analysis
  10. Analysis of errors
  11. Findings
  12. Limitations
  13. Future Work
  14. Dataset
  15. Conclusion
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Examples for errors in cultural symbolism.
  • Figure 2: Examples for errors in absence reasoning.
  • Figure 3: Examples for errors in single-skill fixation.
  • Figure 4: Reasoning quality metrics by model and prompting strategy. CG prompting improves completeness and cognitive skill use across all models, even when correctness gains are modest.