AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?

TL;DR

This paper tackles the challenge of evaluating large vision-language models in a flexible, on-demand manner. It introduces AutoBench-V, an automated framework that uses LVLM examiners and text-to-image generation to create diverse, controllable visual benchmarks across multiple capability areas, while implementing self-validation and bias-mitigation mechanisms. Through extensive experiments with nine LVLMs, the authors show that performance deteriorates with task difficulty and reveal model-specific strengths and weaknesses, supported by ablations and human evaluation. The work offers a scalable, low-cost paradigm for multimodal benchmarking with robust bias controls and error-checking, paving the way for more dynamic and reliable LVLM assessment.

Abstract

Large Vision-Language Models (LVLMs) have become essential for advancing the integration of visual and linguistic information. However, the evaluation of LVLMs presents significant challenges as the evaluation benchmark always demands lots of human cost for its construction, and remains static, lacking flexibility once constructed. Even though automatic evaluation has been explored in textual modality, the visual modality remains under-explored. As a result, in this work, we address a question: "Can LVLMs themselves be used to benchmark each other in the visual automatically domain?". We introduce AutoBench-V, an automated framework for serving evaluation on demand, i.e., benchmarking LVLMs based on specific aspects of model capability. AutoBench-V leverages text-to-image models to generate relevant image samples and then utilizes LVLMs to orchestrate visual question-answering (VQA) tasks, completing the evaluation process efficiently and flexibly. Through an extensive evaluation of nine popular LVLMs across five demanded user inputs (i.e., evaluation capabilities), the framework shows effectiveness and reliability.

Figures (32)

• Figure 1: An overview of the AutoBench-V pipeline.
• Figure 2: Five key evaluation dimensions in AutoBench-V—Basic Understanding, Reasoning Capacity, Semantic Understanding, Spatial Understanding, and Atmospheric Understanding—and their sub-aspects, illustrated by the representative question–image pairs.
• Figure 3: A comprehensive overview of the AutoBench-V framework.
• Figure 4: Models' accuracy (here we take "basic understanding" as an example) before and after the option adjustment, without image input (only guess the answer by textual information).
• Figure 5: Comparison of answer distribution under position bias conditions. Correct answers at A v.s. correct answers are evenly distributed at A,B,C,D.