Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improve MLLM Benchmark Efficiency through Interview

Farong Wen, Yijin Guo, Junying Wang, Jiaohao Xiao, Yingjie Zhou, Ye Shen, Qi Jia, Chunyi Li, Zicheng Zhang

TL;DR

The paper tackles the inefficiency of static, full-coverage benchmarks for Multimodal LLMs by introducing MITV, an interview-style adaptive evaluation that probes abilities with a minimal question set. MITV builds a difficulty-labeled dataset by fusing existing benchmarks and employs a three-module pipeline—dynamic question selection, an MLLM interviewer, and result evaluation—driven by an information-theoretic criterion that targets $p_l \approx 0.5$. The approach is validated on 19 MLLMs across four benchmarks, achieving higher rank correlations (SRCC, PLCC, KRCC) with far fewer questions than random sampling, demonstrating generalization and efficiency. The results indicate MITV can provide fast, scalable, and reliable MLLM assessment suitable for both research and deployment contexts, with practical guidance for future benchmark design and evaluation pipelines.

Abstract

The rapid development of Multimodal Large Language Models (MLLM) has led to a wide range of MLLM applications, and a number of benchmark datasets have sprung up in order to assess MLLM abilities. However, full-coverage Q&A testing on large-scale data is resource-intensive and time-consuming. To address this issue, we propose the MLLM Interview (MITV) strategy, which aims to quickly obtain MLLM performance metrics by quizzing fewer question. First, First, we constructed the interview dataset, which was built on an existing MLLM assessment dataset, by adding difficulty labels based on the performance of some typical MLLMs in this dataset. Second, we propose an MLLM Interview strategy, which obtains an initial performance situation of the large model by quizzing a small number of topics and then continuously tries to test the model's limits. Through extensive experiments, the result shows that the MITV strategy proposed in this paper performs well on MLLM benchmark datasets, and it is able to obtain the model evaluation capability faster through a small number of questions and answers.

Improve MLLM Benchmark Efficiency through Interview

TL;DR

The paper tackles the inefficiency of static, full-coverage benchmarks for Multimodal LLMs by introducing MITV, an interview-style adaptive evaluation that probes abilities with a minimal question set. MITV builds a difficulty-labeled dataset by fusing existing benchmarks and employs a three-module pipeline—dynamic question selection, an MLLM interviewer, and result evaluation—driven by an information-theoretic criterion that targets . The approach is validated on 19 MLLMs across four benchmarks, achieving higher rank correlations (SRCC, PLCC, KRCC) with far fewer questions than random sampling, demonstrating generalization and efficiency. The results indicate MITV can provide fast, scalable, and reliable MLLM assessment suitable for both research and deployment contexts, with practical guidance for future benchmark design and evaluation pipelines.

Abstract

The rapid development of Multimodal Large Language Models (MLLM) has led to a wide range of MLLM applications, and a number of benchmark datasets have sprung up in order to assess MLLM abilities. However, full-coverage Q&A testing on large-scale data is resource-intensive and time-consuming. To address this issue, we propose the MLLM Interview (MITV) strategy, which aims to quickly obtain MLLM performance metrics by quizzing fewer question. First, First, we constructed the interview dataset, which was built on an existing MLLM assessment dataset, by adding difficulty labels based on the performance of some typical MLLMs in this dataset. Second, we propose an MLLM Interview strategy, which obtains an initial performance situation of the large model by quizzing a small number of topics and then continuously tries to test the model's limits. Through extensive experiments, the result shows that the MITV strategy proposed in this paper performs well on MLLM benchmark datasets, and it is able to obtain the model evaluation capability faster through a small number of questions and answers.

Paper Structure

This paper contains 12 sections, 4 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction and Related Works
  2. Dataset Construction
  3. Dataset Preparation
  4. Difficulty Calculation
  5. Proposed Method
  6. Difficulty Determination
  7. MLLM Interview Module
  8. Data Process Module
  9. Experiment
  10. Experiment Detail
  11. Performance Discussion
  12. Conclusion

Figures (3)

  • Figure 1: Motivation for our work. Inspired by human interview processes, we propose an interview strategy that dynamically adjusts questions based on MLLM performance, achieving more effective rankings than random sampling with the same number of questions.
  • Figure 2: Difficulty distribution of questions in different benchmarks.
  • Figure 3: The proposed MITV strategy framework.