Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PM4Bench: Benchmarking Large Vision-Language Models with Parallel Multilingual Multi-Modal Multi-task Corpus

Junyuan Gao, Jiahe Song, Jiang Wu, Runchuan Zhu, Guanlin Shen, Shasha Wang, Xingjian Wei, Haote Yang, Songyang Zhang, Weijia Li, Bin Wang, Dahua Lin, Lijun Wu, Conghui He

TL;DR

PM4Bench tackles fair evaluation of multilingual LVLMs by introducing a strictly parallel multilingual multimodal benchmark across 10 languages and three tasks (MDUR, MIQA, MSOCR) with both traditional and vision input settings. It reveals that the vision setting substantially worsens performance and amplifies cross-lingual disparities, with OCR robustness emerging as a key bottleneck. Through a comprehensive evaluation of 10 LVLMs, PM4Bench shows that model scaling mitigates some cross-lingual gaps in vision, but OCR and script diversity remain central challenges. The work provides a practical, open benchmark for driving advances in multilingual OCR, cross-lingual alignment, and robust multimodal reasoning in LVLMs, with direct implications for real-world, language-diverse AI agents.

Abstract

While Large Vision-Language Models (LVLMs) demonstrate promising multilingual capabilities, their evaluation is currently hindered by two critical limitations: (1) the use of non-parallel corpora, which conflates inherent language capability gaps with dataset artifacts, precluding a fair assessment of cross-lingual alignment; and (2) disjointed multimodal inputs, which deviate from real-world scenarios where most texts are embedded within visual contexts. To address these challenges, we propose PM4Bench, the first Multilingual Multi-Modal Multi-task Benchmark constructed on a strictly parallel corpus across 10 languages. By eliminating content divergence, our benchmark enables a fair comparison of model capabilities across different languages. We also introduce a vision setting where textual queries are visually fused into images, compelling models to jointly "see," "read," and "think". Extensive evaluation of 10 LVLMs uncover a substantial performance drop in the Vision setting compared to standard inputs. Further analysis reveals that OCR capability is not only a general bottleneck but also contributes to cross-lingual performance disparities, suggesting that improving multilingual OCR is essential for advancing LVLM performance. We will release PM4Bench at https://github.com/opendatalab/PM4Bench .

PM4Bench: Benchmarking Large Vision-Language Models with Parallel Multilingual Multi-Modal Multi-task Corpus

TL;DR

PM4Bench tackles fair evaluation of multilingual LVLMs by introducing a strictly parallel multilingual multimodal benchmark across 10 languages and three tasks (MDUR, MIQA, MSOCR) with both traditional and vision input settings. It reveals that the vision setting substantially worsens performance and amplifies cross-lingual disparities, with OCR robustness emerging as a key bottleneck. Through a comprehensive evaluation of 10 LVLMs, PM4Bench shows that model scaling mitigates some cross-lingual gaps in vision, but OCR and script diversity remain central challenges. The work provides a practical, open benchmark for driving advances in multilingual OCR, cross-lingual alignment, and robust multimodal reasoning in LVLMs, with direct implications for real-world, language-diverse AI agents.

Abstract

While Large Vision-Language Models (LVLMs) demonstrate promising multilingual capabilities, their evaluation is currently hindered by two critical limitations: (1) the use of non-parallel corpora, which conflates inherent language capability gaps with dataset artifacts, precluding a fair assessment of cross-lingual alignment; and (2) disjointed multimodal inputs, which deviate from real-world scenarios where most texts are embedded within visual contexts. To address these challenges, we propose PM4Bench, the first Multilingual Multi-Modal Multi-task Benchmark constructed on a strictly parallel corpus across 10 languages. By eliminating content divergence, our benchmark enables a fair comparison of model capabilities across different languages. We also introduce a vision setting where textual queries are visually fused into images, compelling models to jointly "see," "read," and "think". Extensive evaluation of 10 LVLMs uncover a substantial performance drop in the Vision setting compared to standard inputs. Further analysis reveals that OCR capability is not only a general bottleneck but also contributes to cross-lingual performance disparities, suggesting that improving multilingual OCR is essential for advancing LVLM performance. We will release PM4Bench at https://github.com/opendatalab/PM4Bench .

Paper Structure

This paper contains 28 sections, 17 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. PM4Bench
  4. Design Principles
  5. Language Selection
  6. Task Introduction
  7. Translation Pipeline
  8. Experiments
  9. Evaluated models
  10. Inference Configuration
  11. Task settings
  12. Metrics
  13. Results & Findings
  14. Overall performance & cross-lingual disparity
  15. Vision setting is more challenging
  16. ...and 13 more sections

Figures (17)

  • Figure 1: PM4Bench includes parallel corpora in 10 languages and features two settings: traditional and vision, with 3 tasks: MDUR, MIQA, and MSOCR. Leveraging PM4Bench, we are able to comprehensively evaluate the multidimensional capabilities of LVLMs under multimodal × multilingual scenarios and conduct a fair comparison and in-depth analysis of model performance across different languages.
  • Figure 2: The translation of PM4Bench's parallel texts involves three steps: 1) Kimi K2 Reference Generation; 2) Human Rewriting; 3) Claude Post-Selection.
  • Figure 3: Comparison between traditional and vision setting
  • Figure 4: Scores (vision setting) of each model, each language.
  • Figure 5: Performance shift from traditional to vision setting, where Usefulness is the average score on MDUR and MIQA.
  • ...and 12 more figures