Information Density Principle for MLLM Benchmarks

TL;DR

The paper introduces the Information Density principle as an entropy-based framework to quantify the informativeness of multimodal benchmarks for MLLMs. It decomposes benchmark quality into four independent dimensions—Fallacy, Difficulty, Redundancy, and Diversity—and formalizes their relation to overall information via $E(I) \propto (1 - D_{fal}) \cdot D_{dif} \cdot (1 - D_{red}) \cdot D_{div}$. The authors validate the approach by auditing 19 mainstream benchmarks (17,912+ samples) with a Human-Model-Data evaluation pipeline and demonstrate strong alignment between automated metrics and human judgment. They find that recent benchmarks improve on some dimensions but still lag in Diversity, highlighting a need for balanced, high-information benchmarks to better guide MLLM development and benchmarking practices.

Abstract

With the emergence of Multimodal Large Language Models (MLLMs), hundreds of benchmarks have been developed to ensure the reliability of MLLMs in downstream tasks. However, the evaluation mechanism itself may not be reliable. For developers of MLLMs, questions remain about which benchmark to use and whether the test results meet their requirements. Therefore, we propose a critical principle of Information Density, which examines how much insight a benchmark can provide for the development of MLLMs. We characterize it from four key dimensions: (1) Fallacy, (2) Difficulty, (3) Redundancy, (4) Diversity. Through a comprehensive analysis of more than 10,000 samples, we measured the information density of 19 MLLM benchmarks. Experiments show that using the latest benchmarks in testing can provide more insight compared to previous ones, but there is still room for improvement in their information density. We hope this principle can promote the development and application of future MLLM benchmarks. Project page: https://github.com/lcysyzxdxc/bench4bench

Figures (11)

• Figure 1: The significance of the information density for the MLLM research community, which is a principle for selecting suitable benchmarks. Validating MLLMs on a high information density benchmark can provide insight for their future development; However, low information density benchmarks give less useful information, or the information is not reliable, due to the four defects above.
• Figure 2: Fallacy cases for the benchmark original label. For the original label and other options, excepting the label is correct and others are incorrect, the other three correctness situations make up the cases. [Keys: Sub-type, Definition]
• Figure 3: Redundancy cases for the MLLM answer. If MLLM can give correct inference when image/text information is invisible, then this discarded part is redundant. [Keys: Sub-type, Definition, Discarded infomation]
• Figure 4: Image Diversity in Data Eval, consists of five low-level attributes. Their distribution is merged to match the Model Eval. Benchmarks with a wide distribution have strong diversity, while benchmarks with a flat distribution tend to have similar content.
• Figure 5: Text Diversity in Data Eval, consists of ten first-word question formats. Their ratio is merged to match the Model Eval. Benchmarks with multiple question formats have greater diversity, while benchmarks asking the same format need improvement.