ComprehendEdit: A Comprehensive Dataset and Evaluation Framework for Multimodal Knowledge Editing

TL;DR

ComprehendEdit tackles the problem of updating multimodal knowledge by introducing a comprehensive eight-task benchmark and two in-domain metrics, Knowledge Generalization Index (KGI) and Knowledge Preservation Index (KPI), which assess how edits generalize to in-domain misclassifications and preserve correct knowledge within the same domain without relying on AI-synthesized content. It proposes Hierarchical In-Context Editing (HICE), a two-stage method that uses a text-feature classifier and memory-based demonstrations to decide when to apply edits, with a ridge-based weight $W^*$ and memory modules $M_1$ and $M_2$ guiding in-domain updates. Experimental results reveal that existing editing methods often struggle to balance reliability, generality, locality, and in-domain preservation, while HICE achieves competitive traditional metrics and superior KPI/KGI on ComprehendEdit across multiple MLLMs. Overall, the framework provides a robust, diverse evaluation foundation and a practical baseline for advancing robust, domain-aware multimodal knowledge editing.

Abstract

Large multimodal language models (MLLMs) have revolutionized natural language processing and visual understanding, but often contain outdated or inaccurate information. Current multimodal knowledge editing evaluations are limited in scope and potentially biased, focusing on narrow tasks and failing to assess the impact on in-domain samples. To address these issues, we introduce ComprehendEdit, a comprehensive benchmark comprising eight diverse tasks from multiple datasets. We propose two novel metrics: Knowledge Generalization Index (KGI) and Knowledge Preservation Index (KPI), which evaluate editing effects on in-domain samples without relying on AI-synthetic samples. Based on insights from our framework, we establish Hierarchical In-Context Editing (HICE), a baseline method employing a two-stage approach that balances performance across all metrics. This study provides a more comprehensive evaluation framework for multimodal knowledge editing, reveals unique challenges in this field, and offers a baseline method demonstrating improved performance. Our work opens new perspectives for future research and provides a foundation for developing more robust and effective editing techniques for MLLMs. The ComprehendEdit benchmark and implementation code are available at https://github.com/yaohui120/ComprehendEdit.

Figures (9)

• Figure 1: Concept of Multimodal Knowledge Editing. The framework is to correct the wrong answer for the editing sample ("Eagle" to "Parrot") while maintaining the output for unrelated samples ("2" to "2").
• Figure 2: Knowledge Distortion in Multimodal Knowledge Editing. It shows how the model maintains correct outputs for out-of-domain samples but struggles with in-domain samples, highlighting the challenge in preserving and generalizing knowledge.
• Figure 3: Performance comparison of knowledge editing methods on E-VQA benchmark. The range of values for Rel, T-G, T-L, M-L on two backbones are [0, 100], while the ranges of values for I-KGI, T-KGI, I-KPI, T-KPI are [0, 16] on BLIP-2 OPT and [0, 25] on MiniGPT-4.
• Figure 4: Performance comparison of knowledge editing methods on ComprehendEdit benchmark. The range of values for Rel, T-G, T-L, M-L on two backbones is [0, 100], while the range of values for I-KGI, T-KGI, I-KPI, T-KPI is [0, 40].
• Figure 5: Illustration of constructing classifier $W^*$. We first exact features of questions by pre-trained model $h$, and then project these features to obtain $F_p$. $F_p$ are used to calculate $W^*$ by Eq. (11).