Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Human-Aligned Bench: Fine-Grained Assessment of Reasoning Ability in MLLMs vs. Humans

Yansheng Qiu, Li Xiao, Zhaopan Xu, Pengfei Zhou, Zheng Wang, Kaipeng Zhang

TL;DR

This work introduces the Human-Aligned Bench to rigorously assess how well multimodal language models align with human reasoning on context-driven tasks. By compiling 9,794 bilingual questions across four reasoning types drawn from civil service exams and pairing each with human success data and common missteps, the benchmark enables fine-grained analysis of model reasoning versus human cognition. Extensive experiments across open-source and proprietary MLLMs reveal persistent gaps, especially in visual reasoning, and uncover phenomena such as fake or prompt-dependent reasoning. The study provides actionable insights to guide the development of next-generation multimodal models and strengthens the methodology for evaluating true reasoning versus memorization.

Abstract

The goal of achieving Artificial General Intelligence (AGI) is to imitate humans and surpass them. Models such as OpenAI's o1, o3, and DeepSeek's R1 have demonstrated that large language models (LLMs) with human-like reasoning capabilities exhibit exceptional performance and are being gradually integrated into multimodal large language models (MLLMs). However, whether these models possess capabilities comparable to humans in handling reasoning tasks remains unclear at present. In this paper, we propose Human-Aligned Bench, a benchmark for fine-grained alignment of multimodal reasoning with human performance. Specifically, we collected 9,794 multimodal questions that solely rely on contextual reasoning, including bilingual (Chinese and English) multimodal questions and pure text-based questions, encompassing four question types: visual reasoning, definition judgment, analogical reasoning, and logical judgment. More importantly, each question is accompanied by human success rates and options that humans are prone to choosing incorrectly. Extensive experiments on the Human-Aligned Bench reveal notable differences between the performance of current MLLMs in multimodal reasoning and human performance. The findings on our benchmark provide insights into the development of the next-generation models.

Human-Aligned Bench: Fine-Grained Assessment of Reasoning Ability in MLLMs vs. Humans

TL;DR

This work introduces the Human-Aligned Bench to rigorously assess how well multimodal language models align with human reasoning on context-driven tasks. By compiling 9,794 bilingual questions across four reasoning types drawn from civil service exams and pairing each with human success data and common missteps, the benchmark enables fine-grained analysis of model reasoning versus human cognition. Extensive experiments across open-source and proprietary MLLMs reveal persistent gaps, especially in visual reasoning, and uncover phenomena such as fake or prompt-dependent reasoning. The study provides actionable insights to guide the development of next-generation multimodal models and strengthens the methodology for evaluating true reasoning versus memorization.

Abstract

The goal of achieving Artificial General Intelligence (AGI) is to imitate humans and surpass them. Models such as OpenAI's o1, o3, and DeepSeek's R1 have demonstrated that large language models (LLMs) with human-like reasoning capabilities exhibit exceptional performance and are being gradually integrated into multimodal large language models (MLLMs). However, whether these models possess capabilities comparable to humans in handling reasoning tasks remains unclear at present. In this paper, we propose Human-Aligned Bench, a benchmark for fine-grained alignment of multimodal reasoning with human performance. Specifically, we collected 9,794 multimodal questions that solely rely on contextual reasoning, including bilingual (Chinese and English) multimodal questions and pure text-based questions, encompassing four question types: visual reasoning, definition judgment, analogical reasoning, and logical judgment. More importantly, each question is accompanied by human success rates and options that humans are prone to choosing incorrectly. Extensive experiments on the Human-Aligned Bench reveal notable differences between the performance of current MLLMs in multimodal reasoning and human performance. The findings on our benchmark provide insights into the development of the next-generation models.
Paper Structure (31 sections, 4 figures, 6 tables)

This paper contains 31 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Human-Aligned Bench
  4. Experiments and Results
  5. Experimental Setup
  6. Overall Result
  7. Compare with Human Ability
  8. Fake Reasoning Analysis
  9. Conclusion and Limitation
  10. Additional Results
  11. Global Results
  12. MLLMs VS. LLM on Text Quesitons
  13. Prompts
  14. Prompt for Creating English Questions
  15. Prompt for Creating Chinese Questions
  16. ...and 16 more sections

Figures (4)

  • Figure 1: Overview of Human-Aligned Bench. Human-Aligned Bench contains 4 categories of questions, each of which has both Chinese and English versions. Each question contains human scoring rates and error-prone options. These questions require models’ abilities in visual logic and pure text reasoning.
  • Figure 2: Data curation and statistics of our Human-Aligned Bench. The data curation pipeline consists of four stages: data collection, screening, parsing, and processing.
  • Figure 3: Performance (%) on the Human-Aligned Bench across different modalities (text, image) and bilingual (Chinese and English ) in five human difficulty levels. (a) delineates the evaluation metrics for MLLMs, showcasing their mean accuracy across different modalities and tiers of difficulty. (b) presents the evaluation metrics for MLLMs, illustrating their mean accuracy across different languages and levels of complexity.
  • Figure 4: Consistency on Response and Error. (a), (b), and (c) represent the Consistency on Response between MLLMs and humans. (d), (e), and (f) denote the Consistency on Error between MLLMs and humans. In the heatmap, each position (x, y) represents the proportion of consistency between the results of prediction x and prediction y across all responses/all errors.