Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Percept-V Challenge: Can Multimodal LLMs Crack Simple Perception Problems?

Samrajnee Ghosh, Naman Agarwal, Hemanshu Garg, Chinmay Mittal, Mausam, Parag Singla

TL;DR

The paper introduces Percept-V, a $6{,}000$-image, automatically generated dataset spanning $30$ domains to evaluate pure visual perception in multimodal LLMs via TVPS-4 skills. Across six state-of-the-art models, results reveal strong gaps relative to human performance, with model accuracy degrading as problem size increases and performance remaining inconsistent across skills. The study includes a human benchmark showing humans outperform models by a substantial margin, highlighting perceptual limitations in current MLLMs. The authors advocate using Percept-V as a standard, perception-focused benchmark to guide the development of more robust perceptual capabilities in future multimodal systems.

Abstract

Cognitive science research treats visual perception, the ability to understand and make sense of a visual input, as one of the early developmental signs of intelligence. Its TVPS-4 framework categorizes and tests human perception into seven skills such as visual discrimination, and form constancy. Do Multimodal Large Language Models (MLLMs) match up to humans in basic perception? Even though there are many benchmarks that evaluate MLLMs on advanced reasoning and knowledge skills, there is limited research that focuses evaluation on simple perception. In response, we introduce Percept-V, a dataset containing 6000 program-generated uncontaminated images divided into 30 domains, where each domain tests one or more TVPS-4 skills. Our focus is on perception, so we make our domains quite simple and the reasoning and knowledge required for solving them are minimal. Since modern-day MLLMs can solve much more complex tasks, our a-priori expectation is that they will solve these domains very easily. Contrary to our belief, our experiments show a weak performance of SoTA proprietary and open-source MLLMs compared to very high human performance on Percept-V. We find that as number of objects in the image increases, performance goes down rather fast. Our experiments also identify the perception skills that are considerably harder for all models.

The Percept-V Challenge: Can Multimodal LLMs Crack Simple Perception Problems?

TL;DR

The paper introduces Percept-V, a -image, automatically generated dataset spanning domains to evaluate pure visual perception in multimodal LLMs via TVPS-4 skills. Across six state-of-the-art models, results reveal strong gaps relative to human performance, with model accuracy degrading as problem size increases and performance remaining inconsistent across skills. The study includes a human benchmark showing humans outperform models by a substantial margin, highlighting perceptual limitations in current MLLMs. The authors advocate using Percept-V as a standard, perception-focused benchmark to guide the development of more robust perceptual capabilities in future multimodal systems.

Abstract

Cognitive science research treats visual perception, the ability to understand and make sense of a visual input, as one of the early developmental signs of intelligence. Its TVPS-4 framework categorizes and tests human perception into seven skills such as visual discrimination, and form constancy. Do Multimodal Large Language Models (MLLMs) match up to humans in basic perception? Even though there are many benchmarks that evaluate MLLMs on advanced reasoning and knowledge skills, there is limited research that focuses evaluation on simple perception. In response, we introduce Percept-V, a dataset containing 6000 program-generated uncontaminated images divided into 30 domains, where each domain tests one or more TVPS-4 skills. Our focus is on perception, so we make our domains quite simple and the reasoning and knowledge required for solving them are minimal. Since modern-day MLLMs can solve much more complex tasks, our a-priori expectation is that they will solve these domains very easily. Contrary to our belief, our experiments show a weak performance of SoTA proprietary and open-source MLLMs compared to very high human performance on Percept-V. We find that as number of objects in the image increases, performance goes down rather fast. Our experiments also identify the perception skills that are considerably harder for all models.

Paper Structure

This paper contains 46 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset
  4. Experiments
  5. Main Results
  6. Domain-wise Analysis:
  7. Skill-wise Analysis:
  8. Performance vs Size
  9. Human Study
  10. Compliance with Output Format
  11. Conclusion
  12. Appendix
  13. Domain examples of MLLMs
  14. change color
  15. circle boxes
  16. ...and 31 more sections

Figures (3)

  • Figure 1: Sample questions from the Percept-V dataset illustrating tasks related to different visual perception skills as defined by the TVPS-4 framework. Each example shows the image prompt, the correct answer, and a typical incorrect response from an MLLM.
  • Figure 2: The overall accuracy of all models in different skills.
  • Figure 3: Comparison of MLLM average accuracy versus human accuracy across the seven TVPS-4 skills.