Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WiseEdit: Benchmarking Cognition- and Creativity-Informed Image Editing

Kaihang Pan, Weile Chen, Haiyi Qiu, Qifan Yu, Wendong Bu, Zehan Wang, Yun Zhu, Juncheng Li, Siliang Tang

TL;DR

WiseEdit introduces a knowledge-intensive benchmark for cognition- and creativity-informed image editing. It decomposes editing into Awareness, Interpretation, and Imagination, with a WiseEdit-Complex variant, and covers declarative, procedural, and metacognitive knowledge across diverse domains in English and Chinese. It features 1,220 test cases and an automatic evaluation pipeline (GPT-4o) assessing Instruction Following, Detail Preserving, Visual Quality, Knowledge Fidelity, and Creative Fusion across 21 models (17 open-source, 4 closed-source). Experiments reveal persistent gaps in knowledge-based reasoning and compositional creativity, highlighting the need for better multi-image handling and cross-lingual instruction following.

Abstract

Recent image editing models boast next-level intelligent capabilities, facilitating cognition- and creativity-informed image editing. Yet, existing benchmarks provide too narrow a scope for evaluation, failing to holistically assess these advanced abilities. To address this, we introduce WiseEdit, a knowledge-intensive benchmark for comprehensive evaluation of cognition- and creativity-informed image editing, featuring deep task depth and broad knowledge breadth. Drawing an analogy to human cognitive creation, WiseEdit decomposes image editing into three cascaded steps, i.e., Awareness, Interpretation, and Imagination, each corresponding to a task that poses a challenge for models to complete at the specific step. It also encompasses complex tasks, where none of the three steps can be finished easily. Furthermore, WiseEdit incorporates three fundamental types of knowledge: Declarative, Procedural, and Metacognitive knowledge. Ultimately, WiseEdit comprises 1,220 test cases, objectively revealing the limitations of SoTA image editing models in knowledge-based cognitive reasoning and creative composition capabilities. The benchmark, evaluation code, and the generated images of each model will be made publicly available soon. Project Page: https://qnancy.github.io/wiseedit_project_page/.

WiseEdit: Benchmarking Cognition- and Creativity-Informed Image Editing

TL;DR

WiseEdit introduces a knowledge-intensive benchmark for cognition- and creativity-informed image editing. It decomposes editing into Awareness, Interpretation, and Imagination, with a WiseEdit-Complex variant, and covers declarative, procedural, and metacognitive knowledge across diverse domains in English and Chinese. It features 1,220 test cases and an automatic evaluation pipeline (GPT-4o) assessing Instruction Following, Detail Preserving, Visual Quality, Knowledge Fidelity, and Creative Fusion across 21 models (17 open-source, 4 closed-source). Experiments reveal persistent gaps in knowledge-based reasoning and compositional creativity, highlighting the need for better multi-image handling and cross-lingual instruction following.

Abstract

Recent image editing models boast next-level intelligent capabilities, facilitating cognition- and creativity-informed image editing. Yet, existing benchmarks provide too narrow a scope for evaluation, failing to holistically assess these advanced abilities. To address this, we introduce WiseEdit, a knowledge-intensive benchmark for comprehensive evaluation of cognition- and creativity-informed image editing, featuring deep task depth and broad knowledge breadth. Drawing an analogy to human cognitive creation, WiseEdit decomposes image editing into three cascaded steps, i.e., Awareness, Interpretation, and Imagination, each corresponding to a task that poses a challenge for models to complete at the specific step. It also encompasses complex tasks, where none of the three steps can be finished easily. Furthermore, WiseEdit incorporates three fundamental types of knowledge: Declarative, Procedural, and Metacognitive knowledge. Ultimately, WiseEdit comprises 1,220 test cases, objectively revealing the limitations of SoTA image editing models in knowledge-based cognitive reasoning and creative composition capabilities. The benchmark, evaluation code, and the generated images of each model will be made publicly available soon. Project Page: https://qnancy.github.io/wiseedit_project_page/.

Paper Structure

This paper contains 41 sections, 20 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Image Editing Models.
  4. Image Editing Benchmarks.
  5. WiseEdit
  6. Benchmark Construction
  7. Task Depth.
  8. Task Format.
  9. Knowledge-Intensive Evaluation in WiseEdit
  10. Evaluation Pipeline
  11. Experiments
  12. Experimental Setup.
  13. Main Results on WiseEdit
  14. Main Results on WiseEdit-Complex
  15. In-Depth Analysis
  16. ...and 26 more sections

Figures (20)

  • Figure 1: (a) Capabilities of different image editing models across varying skill levels. (b) Comparison between hard cases from WiseEdit and easy cases from existing simplistic image editing benchmarks.
  • Figure 2: Taxonomy of WiseEdit (WiseEdit-Complex).
  • Figure 3: Examples of each task type in WiseEdit and WiseEdit-Complex.
  • Figure 4: Qualitative comparison across AnyEdit, Bagel, Qwen-Image-Edit, GPT-Image-1, and Nano Banana.
  • Figure 5: (a) Pearson correlation between human ratings and VLM scores. (b) Performance visualization across different metrics and knowledge-informed cases. (c) Average performance for single- and multi-image inputs. (d) Average performance with and w/o thinking.
  • ...and 15 more figures