Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Cognitive Evaluation Benchmark of Image Reasoning and Description for Large Vision-Language Models

Xiujie Song, Mengyue Wu, Kenny Q. Zhu, Chunhao Zhang, Yanyi Chen

TL;DR

CogBench introduces a Cookie Theft–inspired benchmark to systematically probe eight high-level cognitive reasoning dimensions in large vision-language models through a dual setup of image descriptions and multiple-choice VQA. By annotating 251 images with entities, eight CoR dimensions, descriptions, and 2577 questions, the dataset enables fine-grained assessment of recognition versus cognition, with GPT-4–assisted generation and manual refinement guiding question creation. Experimental results show substantial gaps between current LVLMs and human cognition, with GPT-4o and other strong models excelling in recognition but lagging in event- and mental-state reasoning, especially under spontaneous description; prompting and directed reasoning modes significantly boost performance. The work argues for CogBench as a challenging, scalable benchmark that highlights specific cognitive gaps in LVLMs and motivates future model improvements in high-level visual reasoning and narrative understanding.

Abstract

Large Vision-Language Models (LVLMs), despite their recent success, are hardly comprehensively tested for their cognitive abilities. Inspired by the prevalent use of the Cookie Theft task in human cognitive tests, we propose a novel evaluation benchmark to evaluate high-level cognitive abilities of LVLMs using images with rich semantics. The benchmark consists of 251 images along with comprehensive annotations. It defines eight reasoning capabilities and comprises an image description task and a visual question answering task. Our evaluation of well-known LVLMs shows that there is still a significant gap in cognitive abilities between LVLMs and humans.

A Cognitive Evaluation Benchmark of Image Reasoning and Description for Large Vision-Language Models

TL;DR

CogBench introduces a Cookie Theft–inspired benchmark to systematically probe eight high-level cognitive reasoning dimensions in large vision-language models through a dual setup of image descriptions and multiple-choice VQA. By annotating 251 images with entities, eight CoR dimensions, descriptions, and 2577 questions, the dataset enables fine-grained assessment of recognition versus cognition, with GPT-4–assisted generation and manual refinement guiding question creation. Experimental results show substantial gaps between current LVLMs and human cognition, with GPT-4o and other strong models excelling in recognition but lagging in event- and mental-state reasoning, especially under spontaneous description; prompting and directed reasoning modes significantly boost performance. The work argues for CogBench as a challenging, scalable benchmark that highlights specific cognitive gaps in LVLMs and motivates future model improvements in high-level visual reasoning and narrative understanding.

Abstract

Large Vision-Language Models (LVLMs), despite their recent success, are hardly comprehensively tested for their cognitive abilities. Inspired by the prevalent use of the Cookie Theft task in human cognitive tests, we propose a novel evaluation benchmark to evaluate high-level cognitive abilities of LVLMs using images with rich semantics. The benchmark consists of 251 images along with comprehensive annotations. It defines eight reasoning capabilities and comprises an image description task and a visual question answering task. Our evaluation of well-known LVLMs shows that there is still a significant gap in cognitive abilities between LVLMs and humans.
Paper Structure (35 sections, 9 figures, 7 tables)

This paper contains 35 sections, 9 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Dataset Construction
  3. Image Collection
  4. Rule 1: Storytelling
  5. Rule 2: Rich Chain-of-Reasonings
  6. Rule 3: Restricted Content Complexity
  7. Image Annotation
  8. Tasks in CogBench
  9. Image Description Task
  10. Visual Question Answering Task
  11. CogBench Statistics
  12. Experiments
  13. Large Vision-Language Models
  14. CogBench Evaluation Strategy
  15. Evaluation of the Description Task
  16. ...and 20 more sections

Figures (9)

  • Figure 1: Cookie Theft picture description task. The descriptions in the green frame and the orange frame were respectively produced by a healthy 75-year-old man and a 66-year-old woman with probable AD dementia.
  • Figure 2: Comparison between our images and those from previous visual reasoning tasks. Our images contain rich entities and CoRs. Compared to our images, image (a) has fewer entities and CoRs, while image (b) and (c) have some entities but fewer CoRs.
  • Figure 3: An example of the Description task from CogBench.
  • Figure 4: Generating a multiple-choice question based on an [Event Reasoning] CoR annotation.
  • Figure 5: Case study of the Description task. The description is generated by GPT-4o in the Directed Reasoning mode. Recognized entities are marked in blue, and CoRs are marked in green.
  • ...and 4 more figures