MCIE: Multimodal LLM-Driven Complex Instruction Image Editing with Spatial Guidance

TL;DR

This paper tackles the difficulty of performing complex, multi-region instruction edits with high instruction compliance and background consistency. It introduces MCIE-E1, a diffusion-based framework that decomposes complex instructions into sub-instructions using an MLLM and guides edits via Spatial-Aware Cross-Attention and Background-Consistent Cross-Attention. To support this, it presents the MCIE dataset (≈90k samples with semantic-spatial annotations) and CIE-Bench (≈400 evaluation sets with two new metrics: Instruction Compliance and Background Consistency). Experimental results show MCIE-E1 surpasses prior methods, including a notable 23.96% improvement in instruction compliance, across quantitative metrics and human studies, indicating strong potential for real-world, complex instruction-based image editing.

Abstract

Recent advances in instruction-based image editing have shown remarkable progress. However, existing methods remain limited to relatively simple editing operations, hindering real-world applications that require complex and compositional instructions. In this work, we address these limitations from the perspectives of architectural design, data, and evaluation protocols. Specifically, we identify two key challenges in current models: insufficient instruction compliance and background inconsistency. To this end, we propose MCIE-E1, a Multimodal Large Language Model-Driven Complex Instruction Image Editing method that integrates two key modules: a spatial-aware cross-attention module and a background-consistent cross-attention module. The former enhances instruction-following capability by explicitly aligning semantic instructions with spatial regions through spatial guidance during the denoising process, while the latter preserves features in unedited regions to maintain background consistency. To enable effective training, we construct a dedicated data pipeline to mitigate the scarcity of complex instruction-based image editing datasets, combining fine-grained automatic filtering via a powerful MLLM with rigorous human validation. Finally, to comprehensively evaluate complex instruction-based image editing, we introduce CIE-Bench, a new benchmark with two new evaluation metrics. Experimental results on CIE-Bench demonstrate that MCIE-E1 consistently outperforms previous state-of-the-art methods in both quantitative and qualitative assessments, achieving a 23.96% improvement in instruction compliance.

Figures (6)

• Figure 1: We propose MCIE-E1 to address the challenges of weak instruction compliance and background consistency in complex instruction-based image editing.
• Figure 2: Visual results of MCIE-E1. Our method effectively performs complex instruction-based image editing with accurate and consistent outputs.
• Figure 3: (a) shows how multi-turn editing sequences are expanded into multiple complex instruction editing instances. (b) shows the use of Qwen2.5-VL-72B to detect instruction conflicts. (c) illustrates the generation and selection of bounding boxes. (d) compares attention maps and editing results for IP2P and our method during the denoising process.
• Figure 4: Comparison of editing results for sub-instruction bounding boxes with and without the Fourier transform.
• Figure 5: The overall framework of MCIE-E1. It employs an MLLM for instruction decomposition and guiding the diffusion model through two key modules: SACA for enhancing instruction following and BCCA for preserving non-edited regions.