Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MME-Industry: A Cross-Industry Multimodal Evaluation Benchmark

Dongyi Yi, Guibo Zhu, Chenglin Ding, Zongshu Li, Dong Yi, Jinqiao Wang

TL;DR

MME-Industry tackles the gap in evaluating Multimodal Large Language Models across diverse industrial domains by introducing a 1,050-question benchmark spanning 21 sectors, with expert-validated content in English and Chinese. The dataset design deliberately removes OCR-only questions and injects domain-knowledge tasks to stress real-world industrial reasoning, while providing rigorous multilingual evaluation and bias-resistant validation. Through experiments on 10 state-of-the-art MLLMs, the work shows strong cross-language performance for large models (e.g., Qwen2-VL-72B-Instruct achieving the top CN and EN scores) and highlights the critical role of visual context in technical domains, as indicated by substantial no-image score drops. The study lays the groundwork for future benchmarking efforts, emphasizing dataset expansion, broader model coverage, open-source platforms, and continuous evaluation to keep pace with rapid industrial AI advances.

Abstract

With the rapid advancement of Multimodal Large Language Models (MLLMs), numerous evaluation benchmarks have emerged. However, comprehensive assessments of their performance across diverse industrial applications remain limited. In this paper, we introduce MME-Industry, a novel benchmark designed specifically for evaluating MLLMs in industrial settings.The benchmark encompasses 21 distinct domain, comprising 1050 question-answer pairs with 50 questions per domain. To ensure data integrity and prevent potential leakage from public datasets, all question-answer pairs were manually crafted and validated by domain experts. Besides, the benchmark's complexity is effectively enhanced by incorporating non-OCR questions that can be answered directly, along with tasks requiring specialized domain knowledge. Moreover, we provide both Chinese and English versions of the benchmark, enabling comparative analysis of MLLMs' capabilities across these languages. Our findings contribute valuable insights into MLLMs' practical industrial applications and illuminate promising directions for future model optimization research.

MME-Industry: A Cross-Industry Multimodal Evaluation Benchmark

TL;DR

MME-Industry tackles the gap in evaluating Multimodal Large Language Models across diverse industrial domains by introducing a 1,050-question benchmark spanning 21 sectors, with expert-validated content in English and Chinese. The dataset design deliberately removes OCR-only questions and injects domain-knowledge tasks to stress real-world industrial reasoning, while providing rigorous multilingual evaluation and bias-resistant validation. Through experiments on 10 state-of-the-art MLLMs, the work shows strong cross-language performance for large models (e.g., Qwen2-VL-72B-Instruct achieving the top CN and EN scores) and highlights the critical role of visual context in technical domains, as indicated by substantial no-image score drops. The study lays the groundwork for future benchmarking efforts, emphasizing dataset expansion, broader model coverage, open-source platforms, and continuous evaluation to keep pace with rapid industrial AI advances.

Abstract

With the rapid advancement of Multimodal Large Language Models (MLLMs), numerous evaluation benchmarks have emerged. However, comprehensive assessments of their performance across diverse industrial applications remain limited. In this paper, we introduce MME-Industry, a novel benchmark designed specifically for evaluating MLLMs in industrial settings.The benchmark encompasses 21 distinct domain, comprising 1050 question-answer pairs with 50 questions per domain. To ensure data integrity and prevent potential leakage from public datasets, all question-answer pairs were manually crafted and validated by domain experts. Besides, the benchmark's complexity is effectively enhanced by incorporating non-OCR questions that can be answered directly, along with tasks requiring specialized domain knowledge. Moreover, we provide both Chinese and English versions of the benchmark, enabling comparative analysis of MLLMs' capabilities across these languages. Our findings contribute valuable insights into MLLMs' practical industrial applications and illuminate promising directions for future model optimization research.

Paper Structure

This paper contains 16 sections, 1 equation, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Large Language Models
  4. MLLMs Benchmarks
  5. MME-Industry Benchmark
  6. Dataset Construction
  7. Task Categories
  8. Evaluation Metrics
  9. Experiments
  10. Experimental Setup
  11. Model Selection
  12. Parameter Settings
  13. Results and Analysis
  14. Results
  15. Analysis
  16. ...and 1 more sections

Figures (2)

  • Figure 1: MME-Industry: Our benchmark covers over 20 industries. The diagram shows a subset of the MME Industry Evaluation Set, featuring six sectors: power, electronics, chemical, textile, steel, and machinery. Questions are in a multiple-choice format with five options, highlighting the correct answer, representative of the full evaluation set.
  • Figure 2: Comparison of 6 top MLLMs across 21 industries in English evaluate. The maximum score for each industry is 100.