MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge

TL;DR

MMKE-Bench tackles a key gap in multimodal knowledge editing by introducing a free-form, language-grounded representation of knowledge plus three realistic editing types (visual entity, visual semantic, and user-specific). It implements a four-stage construction pipeline and a large, 33-category dataset of 2,940 edited knowledge items with 8,363 images, accompanied by automatically generated and human-verified evaluation questions across reliability, locality, generalization, and portability. Evaluations across three prominent LMMs (BLIP-2, MiniGPT-4, LLaVA-1.5) and five editing methods reveal that no approach dominates all criteria, with visual semantic and user-specific edits proving particularly challenging and sequential editing exposing memory-related limitations. The work thus sets a robust benchmark standard and provides actionable insights to advance robust, real-world multimodal knowledge editing.

Abstract

Knowledge editing techniques have emerged as essential tools for updating the factual knowledge of large language models (LLMs) and multimodal models (LMMs), allowing them to correct outdated or inaccurate information without retraining from scratch. However, existing benchmarks for multimodal knowledge editing primarily focus on entity-level knowledge represented as simple triplets, which fail to capture the complexity of real-world multimodal information. To address this issue, we introduce MMKE-Bench, a comprehensive MultiModal Knowledge Editing Benchmark, designed to evaluate the ability of LMMs to edit diverse visual knowledge in real-world scenarios. MMKE-Bench addresses these limitations by incorporating three types of editing tasks: visual entity editing, visual semantic editing, and user-specific editing. Besides, MMKE-Bench uses free-form natural language to represent and edit knowledge, offering a more flexible and effective format. The benchmark consists of 2,940 pieces of knowledge and 8,363 images across 33 broad categories, with evaluation questions automatically generated and human-verified. We assess five state-of-the-art knowledge editing methods on three prominent LMMs, revealing that no method excels across all criteria, and that visual and user-specific edits are particularly challenging. MMKE-Bench sets a new standard for evaluating the robustness of multimodal knowledge editing techniques, driving progress in this rapidly evolving field.

Figures (28)

• Figure 1: Comparison between the existing benchmark and MMKE-Bench with a detailed example. In this example, the texts in red represent the edited counterfactual content. T/I-Rel represents text and image reliability, T/I-Gen represents text and image generalization and Port represents portability. Previous benchmarks mainly focus on entity recognition editing using a triplet-based knowledge representation format, which does not align with actual scenarios. MMKE-Bench focuses on evaluating diverse semantic editing in realistic scenarios in a natural language format.
• Figure 2: The construction pipeline of MMKE-Bench.
• Figure 3: The types of samples in MMKE-Bench.
• Figure 4: Evaluation comparison of IKE for MiniGPT-4 with existing benchmarks. Port for MMEdit and MIKE, is set 1, as they are not evaluated.
• Figure 5: Case study of editing examples