Redundancy Principles for MLLMs Benchmarks

TL;DR

The paper tackles redundancy in the rapidly expanding field of MLLM benchmarks by proposing a three-faceted framework that quantifies redundancy across benchmark dimensions, test instances, and cross-benchmark comparisons. It defines formal metrics for dimension, instance, and cross-benchmark redundancy using ranking correlations and standard measures (SRCC, PLCC, R^2), and validates the approach with VLMEvalKit data and a detailed MMBench case study. Key findings show that dimensional redundancy is higher for lower-performing models, instance redundancy is pervasive across benchmarks, and cross-benchmark overlap varies by domain (e.g., MathVista vs MathVerse/MathVision), with noise removal improving alignment. The work offers practical design principles to reduce inefficiency, such as optimizing dimension independence, pruning redundant instances, and selecting domain-appropriate benchmarks, ultimately enabling more efficient and informative MLLM evaluations.

Abstract

With the rapid iteration of Multi-modality Large Language Models (MLLMs) and the evolving demands of the field, the number of benchmarks produced annually has surged into the hundreds. The rapid growth has inevitably led to significant redundancy among benchmarks. Therefore, it is crucial to take a step back and critically assess the current state of redundancy and propose targeted principles for constructing effective MLLM benchmarks. In this paper, we focus on redundancy from three key perspectives: 1) Redundancy of benchmark capability dimensions, 2) Redundancy in the number of test questions, and 3) Cross-benchmark redundancy within specific domains. Through the comprehensive analysis over hundreds of MLLMs' performance across more than 20 benchmarks, we aim to quantitatively measure the level of redundancy lies in existing MLLM evaluations, provide valuable insights to guide the future development of MLLM benchmarks, and offer strategies to refine and address redundancy issues effectively. The code is available at https://github.com/zzc-1998/Benchmark-Redundancy.

Figures (9)

• Figure 1: A quick look at the redundancy framework, where (a), (b), and (c) show the general process of computing dimensions redundancy, instances redundancy, and cross-benchmark redundancy respectively.
• Figure 2: Visualizations of dimensions redundancy for MMBench liu2025mmbench on Top-50 and Bottom-50 (marked as 50$^+$ and 50$^-$) MLLMs respectively. More benchmark results can be found in Appendix. \ref{['app:extra']}.
• Figure 3: Bar plots of dimensions redundancy for MMBench liu2025mmbench on Top-50 and Bottom-50 MLLMs. The redundancy values are computed by averaging the redundancy of each dimension with the redundancy of all other dimensions.
• Figure 4: Visualizations of average instance redundancy for (a) Top-50 MLLMs and (b) Bottom-50 MLLMs across 18 LMM benchmarks (A-Bench zhang2024bench, AI2D kembhavi2016diagram, BLINK fu2025blink, HallusionBench guan2023hallusionbench, MMBench liu2025mmbench, MMMU yue2024mmmu, MME fu2024mmecomprehensiveevaluationbenchmark, MMStar chen2024mmstar, MMT ying2024mmt, MMVet yu2023mmvet, OCRBench liu2023ocrbench, Q-Bench wu2023qbenchzhang2024q+, R-Bench-Dis li2024rbench, RealWorldQA RealWorldQA, ScienceQA lu2022learn, SeedBench_IMG li2023seedimg, SeedBench2_Plus li2024seed2plus). Notably, each data point represents the average of 100 sampling iterations to mitigate the impact of randomness.
• Figure 5: Benchmark-specific instance redundancy for (a) Top-50 MLLMs and (b) Bottom-50 MLLMs. The benchmarks include BLINK fu2025blink, ScienceQA lu2022learn, MMMU yue2024mmmu, RealWorldQA RealWorldQA, MMBench liu2025mmbench, MMStar chen2024mmstar, SeedBench_IMG li2023seedimg, and AI2D kembhavi2016diagram. The selection of the Top-50 and Bottom-50 MLLMs is based on the corresponding benchmark.