Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VisualQuest: A Benchmark for Abstract Visual Reasoning in MLLMs

Kelaiti Xiao, Liang Yang, Dongyu Zhang, Paerhati Tulajiang, Hongfei Lin

TL;DR

VisualQuest introduces a 3,551-image benchmark of non-photographic, stylized visuals spanning four knowledge-rich categories to probe abstract visual reasoning in multimodal large language models. Through evaluation of ten leading MLLMs, the study shows that only Gemini-2.5-flash and GPT-4o achieve strong overall performance, with 3.7% of images universally unrecognized, underscoring persistent gaps in multimodal understanding. The results reveal model-specific strengths, such as Gemini’s prowess with stylized public figures and GPT-4o’s linguistic reasoning in visual puns and emoji combinations, while highlighting the need for improved integration of background knowledge and visual reasoning. VisualQuest thus serves as a rigorous resource to drive progress toward more robust, human-aligned abstractive visual reasoning in multimodal systems.

Abstract

We introduce VisualQuest, a novel dataset designed to rigorously evaluate multimodal large language models (MLLMs) on abstract visual reasoning tasks that require the integration of symbolic, cultural, and linguistic knowledge. Unlike existing benchmarks that focus on direct image captioning or classification of realistic images, VisualQuest comprises 3,551 non-photographic, stylized images spanning four categories: Public Figures, Popular Culture, Linguistic Expressions, and Literary Works. Each image is paired with targeted questions to probe complex reasoning. We benchmark ten state-of-the-art MLLMs and find that only Gemini-2.5-flash and GPT-4o achieve strong overall performance, while 3.7 percent of the images remain unrecognized by any model, underscoring persistent challenges in multimodal understanding. Fine-grained analysis shows that Gemini excels at recognizing stylized public figures, whereas GPT-4o leads in linguistic reasoning tasks such as visual puns and emoji combinations. VisualQuest provides a comprehensive and challenging resource for advancing research in abstract visual reasoning and highlights key areas for future model improvement. The dataset is available at https://github.com/xkt88/VISUALQUEST.

VisualQuest: A Benchmark for Abstract Visual Reasoning in MLLMs

TL;DR

VisualQuest introduces a 3,551-image benchmark of non-photographic, stylized visuals spanning four knowledge-rich categories to probe abstract visual reasoning in multimodal large language models. Through evaluation of ten leading MLLMs, the study shows that only Gemini-2.5-flash and GPT-4o achieve strong overall performance, with 3.7% of images universally unrecognized, underscoring persistent gaps in multimodal understanding. The results reveal model-specific strengths, such as Gemini’s prowess with stylized public figures and GPT-4o’s linguistic reasoning in visual puns and emoji combinations, while highlighting the need for improved integration of background knowledge and visual reasoning. VisualQuest thus serves as a rigorous resource to drive progress toward more robust, human-aligned abstractive visual reasoning in multimodal systems.

Abstract

We introduce VisualQuest, a novel dataset designed to rigorously evaluate multimodal large language models (MLLMs) on abstract visual reasoning tasks that require the integration of symbolic, cultural, and linguistic knowledge. Unlike existing benchmarks that focus on direct image captioning or classification of realistic images, VisualQuest comprises 3,551 non-photographic, stylized images spanning four categories: Public Figures, Popular Culture, Linguistic Expressions, and Literary Works. Each image is paired with targeted questions to probe complex reasoning. We benchmark ten state-of-the-art MLLMs and find that only Gemini-2.5-flash and GPT-4o achieve strong overall performance, while 3.7 percent of the images remain unrecognized by any model, underscoring persistent challenges in multimodal understanding. Fine-grained analysis shows that Gemini excels at recognizing stylized public figures, whereas GPT-4o leads in linguistic reasoning tasks such as visual puns and emoji combinations. VisualQuest provides a comprehensive and challenging resource for advancing research in abstract visual reasoning and highlights key areas for future model improvement. The dataset is available at https://github.com/xkt88/VISUALQUEST.

Paper Structure

This paper contains 16 sections, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Compositional and Spatial Visual Reasoning
  4. Abstract, Pattern-Based, and Mathematical Reasoning
  5. Knowledge-Grounded and High-Level Cognitive Reasoning
  6. Dataset Details
  7. Data Collection
  8. Data Curation and Annotation
  9. Dataset Composition
  10. Label structure
  11. Experiments
  12. Evaluated Multimodal Large Language Models
  13. Overall Model Performance
  14. Performance by Background Knowledge Category
  15. Performance by Visual Presentation Method
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Examples from the VisualQuest dataset illustrating the task format. For each input image and question (top two rows), the model must generate a reasoning analysis and the correct label (bottom two rows)
  • Figure 2: Example Set for Training
  • Figure 3: (a) Donut chart showing the distribution of images in the VisualQuest dataset by the number of MLLMs that recognized them. (b) Bar chart comparing the recognition accuracy of ten MLLMs on all images (blue) and after excluding universally unrecognized images (red).
  • Figure 4: Performance of MLLMs on each Category