Resampling Benchmark for Efficient Comprehensive Evaluation of Large Vision-Language Models

TL;DR

Resampling Benchmark for Efficient Comprehensive Evaluation of Large Vision-Language Models addresses the high cost of evaluating VLMs across many benchmarks by introducing ResampledBench, an FPS-based subset that preserves model rankings with a correlation above $0.96$ to full benchmark evaluations while using about $1\%$ of the data. The method relies on a joint image-text feature space and demonstrates that no single benchmark fully covers the evaluation space, while FPS-based sampling reduces redundancy and can mitigate dataset biases. The approach yields practical efficiency gains (≈100×) and improved bias mitigation when filtering established benchmarks like MMStar. Overall, it provides a scalable framework for robust, biased-aware VLM evaluation and points to future work in learning a true multimodal embedding space to further improve benchmarking.

Abstract

We propose an efficient evaluation protocol for large vision-language models (VLMs). Given their broad knowledge and reasoning capabilities, multiple benchmarks are needed for comprehensive assessment, making evaluation computationally expensive. To improve efficiency, we construct a subset that yields results comparable to full benchmark evaluations. Our benchmark classification experiments reveal that no single benchmark fully covers all challenges. We then introduce a subset construction method using farthest point sampling (FPS). Our experiments show that FPS-based benchmarks maintain a strong correlation (> 0.96) with full evaluations while using only ~1\% of the data. Additionally, applying FPS to an existing benchmark improves correlation with overall evaluation results, suggesting its potential to reduce unintended dataset biases.

Figures (8)

• Figure 1: Since it is costly to evaluate VLMs on a large number of accessible benchmarks for the comprehensive evaluation, we sample data from the benchmarks with as fewer samples as possible to make evaluation efficient but as comprehensive as using all the data. As a result, our benchmark, ResampledBench, has approximately 100$\times$ fewer samples than sum of the number of data in the accessible benchmarks, while the model ranks on ResampledBench have a correlation coefficient of over 0.96 with the model ranks averaged over the ranks on all benchmarks.
• Figure 2: Confusion matrices of the benchmark classification experiments for MCQ with image and question input and VQA with image, question, and answer input.
• Figure 3: The correlations between $\mathrm{AvgRank}$ and the ranks on individual benchmarks.
• Figure 4: The correlation coefficient of a various number of data. The broken line denotes the correlation coefficient of AI2D that has the highest correlation coefficient among the existing benchmarks listed in Fig. \ref{['fig:corr_rank']}. Compered to random sampling, FPS achieves higher correlation even with 10$\times$ fewer samples.
• Figure 5: The confusion matrices for MCQ with various inputs. The confusion matrix for MCQ using images and questions is shown in the Sec. 3.