Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MULTI: Multimodal Understanding Leaderboard with Text and Images

Zichen Zhu, Yang Xu, Lu Chen, Jingkai Yang, Yichuan Ma, Yiming Sun, Hailin Wen, Jiaqi Liu, Jinyu Cai, Yingzi Ma, Situo Zhang, Zihan Zhao, Liangtai Sun, Kai Yu

TL;DR

MULTI introduces a large-scale Chinese multimodal benchmark derived from authentic exams to evaluate cross-modal understanding, reasoning, and knowledge recall in MLLMs. It comprises over 18,000 questions (MULTI), plus a challenging 500-item subset (MULTI-Elite) and a 4,596-piece knowledge extension (MULTI-Extend) to probe in-context learning and knowledge transfer. Extensive experiments across 25 models (open- and closed-source) show substantial gaps to human expert baselines, with performance degrading on images, multi-image items, and complex reasoning tasks, highlighting ongoing challenges in cross-modal alignment and reasoning. By providing rich, real-world content and a public evaluation platform, MULTI aims to catalyze the development of expert-level AI for education and beyond.

Abstract

The rapid development of multimodal large language models (MLLMs) raises the question of how they compare to human performance. While existing datasets often feature synthetic or overly simplistic tasks, some models have already surpassed human expert baselines. In this paper, we present MULTI, a Chinese multimodal dataset derived from authentic examination questions. Comprising over 18,000 carefully selected and refined questions, MULTI evaluates models using real-world examination standards, encompassing image-text comprehension, complex reasoning, and knowledge recall. Additionally, We also introduce MULTI-Elite, a 500-question selected hard subset, and MULTI-Extend with more than 4,500 external knowledge context pieces for testing in-context learning capabilities. Our evaluation highlights substantial room for MLLM advancement, with Qwen2-VL-72B achieving a 76.9% accuracy on MULTI and 53.1% on MULTI-Elite leading 25 evaluated models, compared to human expert baselines of 86.1% and 73.1%. MULTI serves not only as a robust evaluation platform but also paves the way for the development of expert-level AI.

MULTI: Multimodal Understanding Leaderboard with Text and Images

TL;DR

MULTI introduces a large-scale Chinese multimodal benchmark derived from authentic exams to evaluate cross-modal understanding, reasoning, and knowledge recall in MLLMs. It comprises over 18,000 questions (MULTI), plus a challenging 500-item subset (MULTI-Elite) and a 4,596-piece knowledge extension (MULTI-Extend) to probe in-context learning and knowledge transfer. Extensive experiments across 25 models (open- and closed-source) show substantial gaps to human expert baselines, with performance degrading on images, multi-image items, and complex reasoning tasks, highlighting ongoing challenges in cross-modal alignment and reasoning. By providing rich, real-world content and a public evaluation platform, MULTI aims to catalyze the development of expert-level AI for education and beyond.

Abstract

The rapid development of multimodal large language models (MLLMs) raises the question of how they compare to human performance. While existing datasets often feature synthetic or overly simplistic tasks, some models have already surpassed human expert baselines. In this paper, we present MULTI, a Chinese multimodal dataset derived from authentic examination questions. Comprising over 18,000 carefully selected and refined questions, MULTI evaluates models using real-world examination standards, encompassing image-text comprehension, complex reasoning, and knowledge recall. Additionally, We also introduce MULTI-Elite, a 500-question selected hard subset, and MULTI-Extend with more than 4,500 external knowledge context pieces for testing in-context learning capabilities. Our evaluation highlights substantial room for MLLM advancement, with Qwen2-VL-72B achieving a 76.9% accuracy on MULTI and 53.1% on MULTI-Elite leading 25 evaluated models, compared to human expert baselines of 86.1% and 73.1%. MULTI serves not only as a robust evaluation platform but also paves the way for the development of expert-level AI.
Paper Structure (48 sections, 4 equations, 16 figures, 13 tables)

This paper contains 48 sections, 4 equations, 16 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Large Language Models (MLLMs).
  4. Benchmarks for MLLMs.
  5. The M[0.8]ULTI Benchmark
  6. Data Construction Process
  7. Data Source
  8. Data Process and Annotation
  9. Statistics
  10. The M[0.8]ULTI-E[0.8]LITE Set
  11. The M[0.8]ULTI-E[0.8]XTEND Background Knowledge Dataset
  12. Comparison with Existing Benchmarks
  13. Experiments
  14. Baselines
  15. Large Models.
  16. ...and 33 more sections

Figures (16)

  • Figure 1: M[0.8]ULTI is a large-scale Chinese multimodal benchmark across multiple domains and aspects.
  • Figure 2: An example of M[0.8]ULTI. English translations of Chinese text are shown for better readability. The markdown format remains as it is.
  • Figure 3: The construction pipeline of M[0.8]ULTI. Upper half: The construction of Multi and Multi-Elite. (1) Collect large-scale open-source data. (2) Select high-quality and evenly distributed open-source Q&As. (3) Collect and transcribe questions from closed-source exams. (4) Convert raw data into markdown format using scripts. (5) Skilled annotators refine, split, and label the data. (6) Assess the difficulty and quality of each question. (7) Further refine and increase difficulty through expert review. (8) Select the most challenging items for the Multi-Elite subset. Lower half: The construction of Multi-Extend. (9) Summarize knowledge from open-source data. (10) Generate additional knowledge via LLMs.
  • Figure 4: An example of the prompt template. It is used when evaluating a multiple-choice question with image context, knowledge piece, and single correct answer.
  • Figure 5: The portion of answer rejection when evaluated on SI set (left) and MI set (right) without image information as input.
  • ...and 11 more figures