CompBench: Benchmarking Complex Instruction-guided Image Editing

TL;DR

This work targets the gap between real-world complex instruction-based image editing and existing benchmarks that emphasize simplistic edits. It introduces CompBench, a large-scale benchmark with 3k+ image-instruction pairs across 9 tasks in 5 categories, and an MLLM-Human collaborative data-generation pipeline with an instruction-decomposition scheme. Through extensive evaluations of 13 models, it demonstrates that current systems struggle with complex scene understanding, multi-object interactions, and reasoning, and it identifies MLLMs and reasoning-driven architectures as pivotal for progress. The authors provide dataset, code, and models to enable future research and emphasize the need for reasoning-aware, controllable editing capabilities.

Abstract

While real-world applications increasingly demand intricate scene manipulation, existing instruction-guided image editing benchmarks often oversimplify task complexity and lack comprehensive, fine-grained instructions. To bridge this gap, we introduce, a large-scale benchmark specifically designed for complex instruction-guided image editing. CompBench features challenging editing scenarios that incorporate fine-grained instruction following, spatial and contextual reasoning, thereby enabling comprehensive evaluation of image editing models' precise manipulation capabilities. To construct CompBench, We propose an MLLM-human collaborative framework with tailored task pipelines. Furthermore, we propose an instruction decoupling strategy that disentangles editing intents into four key dimensions: location, appearance, dynamics, and objects, ensuring closer alignment between instructions and complex editing requirements. Extensive evaluations reveal that CompBench exposes fundamental limitations of current image editing models and provides critical insights for the development of next-generation instruction-guided image editing systems. The dataset, code, and models are available in https://comp-bench.github.io/.

Figures (19)

• Figure 1: Examples of CompBench. The figure showcases diverse instruction-guided image editing tasks across nine categories: object addition, object removal, object replacement, multi-object editing, multi-turn editing, implicit reasoning, action editing, location editing and viewpoint editing).
• Figure 2: Comparison between current datasets or benchmarks and our CompBench.First row: failed cases of other benchmarks. These results fail to maintain background consistencies or introduce noticable artifacts into the editing region. Second row: Examples of other benchmarks. These cases lack scene complexity and instruction comprehensiveness. Third row: Examples of our CompBench. Our benchmark features complex real-world scenarios with precise instructions.
• Figure 3: The construction pipeline of CompBench. The pipeline consists of two main stages: (a) Source data collection and preprocessing, wherein high-quality data are identified through image quality filtering, mask decomposition, occlusion and continuity evaluation, followed by thorough human verification. (b) Task-specific data generation using four specialized pipelines within our MLLM-Human Collaborative Framework, where multimodal large language models generate initial editing instructions that are subsequently validated by humans to ensure high-fidelity, semantically aligned instruction-image pairs for complex editing tasks.
• Figure 4: Characteristics and statistics of CompBench. (a) Task taxonomy of CompBench, illustrating the full range of task types. (b) SSIM ssim comparison among different datasets and benchmarks. Note that UltraEdit ultraedit and InstructPix2pix instructpix2pix are datasets, whereas the remaining entries are benchmarks.
• Figure 5: Overall Model Performance.(a) Top 5 model performace in five major evaluation tasks. (b) Overall model performace across all tasks.