KBE-DME: Dynamic Multimodal Evaluation via Knowledge Enhanced Benchmark Evolution

TL;DR

KBE-DME introduces a graph-based dynamic evaluation framework for multimodal LLMs that evolves static VQA benchmarks by re-selecting key multimodal knowledge triplets and by exploring external knowledge. By modeling each VQA instance as $G_M$, $G_T$, and $G_K$, the method regenerates questions with controllable difficulty, measured by $|E_K|$, through an Extract-Explore-Express pipeline. Extensive experiments on OK-VQA and A-OKVQA across five MLLMs demonstrate that difficulty can be systematically increased via up to three hops, revealing performance degradation that reflects genuine capability gaps while reducing data contamination and saturation risks. The framework is generalizable to diverse multimodal benchmarks and offers a sustainable path for co-evolving evaluation with advancing models.

Abstract

The rapid progress of multimodal large language models (MLLMs) calls for more reliable evaluation protocols. Existing static benchmarks suffer from the potential risk of data contamination and saturation, leading to inflated or misleading performance evaluations. To address these issues, we first apply Graph formulation to represent a static or dynamic VQA sample. With the formulation, we propose Knowledge-enhanced Benchmark Evolution(KBE), a dynamic multimodal evaluation framework. KBE first analyzes the original static benchmark, then expands it by integrating multimodal knowledge, transforming the static benchmark into a controllable, dynamic evolving version. Crucially, KBE can both reconstruct questions by Re-selecting visual information in the original image and expand existing questions with external textual knowledge. It enables difficulty-controllable evaluation by adjusting the degree of question exploration. Extensive experiments demonstrate that KBE alleviates the risk of data contamination, data saturation, and provides a more comprehensive assessment of MLLM capabilities.

Figures (10)

• Figure 1: Figure of the original VQA test sample $S_0$, the perturbed test sample $\tilde{S_0}$ generated by perturbation methods, and the dynamically generated test sample $S_1$ produced by KBE-DME. Here, $G_K$ denotes the key information subgraph extracted from a VQA test sample. As shown, the perturbed VQA test sample does not alter the underlying $G_K$ of the question.
• Figure 2: Figure of our Graph Formulation and the KBE-DME framework. The upper part of the figure uses a static VQA sample $S_0$ to exemplify our graph representation of a VQA problem, while the lower part demonstrates the KBE-DME framework for dynamically constructing VQA test data.
• Figure 3: An example from our data construction process on OK-VQA. $S_0$ denotes the original VQA sample, where $I_0$, $Q_0$, and $A _0$ represent the corresponding image, question, and answer, and $K_0$ is the associated set of key knowledge triples. For the generated data $S_i$ at different hop levels i, we keep the input image unchanged while reconstructing the corresponding questions and answers by altering the composition of the key knowledge triples.
• Figure 4: Graph extraction prompt.
• Figure 5: Key triplets extraction prompt.