Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IQBench: How "Smart'' Are Vision-Language Models? A Study with Human IQ Tests

Tan-Hanh Pham, Phu-Vinh Nguyen, Dang The Hung, Bui Trong Duong, Vu Nguyen Thanh, Chris Ngo, Tri Quang Truong, Truong-Son Hy

TL;DR

IQBench introduces a vision-centric benchmark to assess the fluid intelligence of Vision-Language Models by using standardized visual IQ questions with annotated reasoning patterns. It pairs a dual evaluation framework—Accuracy Score for final answers and Reasoning Score for explanation quality via an LLM judge and human evaluators—with a manually curated dataset of 500 questions across 10 topics to minimize data leakage. Experiments on seven state-of-the-art VLMs reveal that while some models achieve solid accuracy, many struggle with 3D spatial reasoning and anagram tasks, and reasoning quality often does not perfectly align with final predictions. The work emphasizes the importance of evaluating not just what models answer, but how they reason, to push toward more transparent and cognitively capable multimodal systems.

Abstract

Although large Vision-Language Models (VLMs) have demonstrated remarkable performance in a wide range of multimodal tasks, their true reasoning capabilities on human IQ tests remain underexplored. To advance research on the fluid intelligence of VLMs, we introduce **IQBench**, a new benchmark designed to evaluate VLMs on standardized visual IQ tests. We focus on evaluating the reasoning capabilities of VLMs, which we argue are more important than the accuracy of the final prediction. **Our benchmark is visually centric, minimizing the dependence on unnecessary textual content**, thus encouraging models to derive answers primarily from image-based information rather than learned textual knowledge. To this end, we manually collected and annotated 500 visual IQ questions to **prevent unintentional data leakage during training**. Unlike prior work that focuses primarily on the accuracy of the final answer, we evaluate the reasoning ability of the models by assessing their explanations and the patterns used to solve each problem, along with the accuracy of the final prediction and human evaluation. Our experiments show that there are substantial performance disparities between tasks, with models such as `o4-mini`, `gemini-2.5-flash`, and `claude-3.7-sonnet` achieving the highest average accuracies of 0.615, 0.578, and 0.548, respectively. However, all models struggle with 3D spatial and anagram reasoning tasks, highlighting significant limitations in current VLMs' general reasoning abilities. In terms of reasoning scores, `o4-mini`, `gemini-2.5-flash`, and `claude-3.7-sonnet` achieved top averages of 0.696, 0.586, and 0.516, respectively. These results highlight inconsistencies between the reasoning processes of the models and their final answers, emphasizing the importance of evaluating the accuracy of the reasoning in addition to the final predictions.

IQBench: How "Smart'' Are Vision-Language Models? A Study with Human IQ Tests

TL;DR

IQBench introduces a vision-centric benchmark to assess the fluid intelligence of Vision-Language Models by using standardized visual IQ questions with annotated reasoning patterns. It pairs a dual evaluation framework—Accuracy Score for final answers and Reasoning Score for explanation quality via an LLM judge and human evaluators—with a manually curated dataset of 500 questions across 10 topics to minimize data leakage. Experiments on seven state-of-the-art VLMs reveal that while some models achieve solid accuracy, many struggle with 3D spatial reasoning and anagram tasks, and reasoning quality often does not perfectly align with final predictions. The work emphasizes the importance of evaluating not just what models answer, but how they reason, to push toward more transparent and cognitively capable multimodal systems.

Abstract

Although large Vision-Language Models (VLMs) have demonstrated remarkable performance in a wide range of multimodal tasks, their true reasoning capabilities on human IQ tests remain underexplored. To advance research on the fluid intelligence of VLMs, we introduce **IQBench**, a new benchmark designed to evaluate VLMs on standardized visual IQ tests. We focus on evaluating the reasoning capabilities of VLMs, which we argue are more important than the accuracy of the final prediction. **Our benchmark is visually centric, minimizing the dependence on unnecessary textual content**, thus encouraging models to derive answers primarily from image-based information rather than learned textual knowledge. To this end, we manually collected and annotated 500 visual IQ questions to **prevent unintentional data leakage during training**. Unlike prior work that focuses primarily on the accuracy of the final answer, we evaluate the reasoning ability of the models by assessing their explanations and the patterns used to solve each problem, along with the accuracy of the final prediction and human evaluation. Our experiments show that there are substantial performance disparities between tasks, with models such as `o4-mini`, `gemini-2.5-flash`, and `claude-3.7-sonnet` achieving the highest average accuracies of 0.615, 0.578, and 0.548, respectively. However, all models struggle with 3D spatial and anagram reasoning tasks, highlighting significant limitations in current VLMs' general reasoning abilities. In terms of reasoning scores, `o4-mini`, `gemini-2.5-flash`, and `claude-3.7-sonnet` achieved top averages of 0.696, 0.586, and 0.516, respectively. These results highlight inconsistencies between the reasoning processes of the models and their final answers, emphasizing the importance of evaluating the accuracy of the reasoning in addition to the final predictions.
Paper Structure (13 sections, 4 figures, 4 tables)

This paper contains 13 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. IQBench
  3. Data Collection and Generation
  4. Data Quality Control
  5. Evaluation Methods
  6. Experiment and Result
  7. Accuracy Evaluation
  8. Reasoning Evaluation
  9. Human Evaluation and Failure Analysis
  10. Conclusion
  11. Prompts for General VLMs
  12. Prompts for Reasoning VLMs
  13. LLM-as-a-judge Prompt

Figures (4)

  • Figure 1: Representative IQ test covering logic, pattern recognition, and spatial reasoning for VLM evaluation. A sample consists of an input image, a question, an answer, and a possible reasoning pattern.
  • Figure 2: Accuracy evaluation of the advanced multimodal models on IQBench.
  • Figure 3: Reasoning evaluation of the advanced multimodal models on IQBench.
  • Figure 4: Examples of VLM reasoning errors compared to human annotations. (a) Correct reasoning with incorrect prediction in a pattern recognition task. (b) Partially correct reasoning with incorrect prediction in a physics problem.