MIND-Edit: MLLM Insight-Driven Editing via Language-Vision Projection

TL;DR

Seeks to bridge the semantic gap between natural language instructions and visual edits in complex scenes. Proposes an end-to-end MIND-Edit framework that combines diffusion models with a multimodal large language model, featuring an instruction optimization strategy and an MLLM insight-driven editing pathway, plus a joint training regime. Key contributions include (i) the instruction optimization module, (ii) the visual-guided editing via MLLM embeddings injected through an IP-Adapter, (iii) a unified joint-training approach, and (iv) extensive experiments showing superior performance on simple and complex edits. The method advances controllable, semantically aligned image editing with better handling of ambiguous user input.

Abstract

Recent advances in AI-generated content (AIGC) have significantly accelerated image editing techniques, driving increasing demand for diverse and fine-grained edits. Despite these advances, existing image editing methods still face challenges in achieving high precision and semantic accuracy in complex scenarios. Recent studies address this issue by incorporating multimodal large language models (MLLMs) into image editing pipelines. However, current MLLM-based methods mainly rely on interpreting textual instructions, leaving the intrinsic visual understanding of large models largely unexplored, thus resulting in insufficient alignment between textual semantics and visual outcomes. To overcome these limitations, we propose MIND-Edit, an end-to-end image-editing framework integrating pretrained diffusion model with MLLM. MIND-Edit introduces two complementary strategies: (1) a text instruction optimization strategy that clarifies ambiguous user instructions based on semantic reasoning from the MLLM, and (2) an MLLM insight-driven editing strategy that explicitly leverages the intrinsic visual understanding capability of the MLLM to infer editing intent and guide the diffusion process via generated visual embeddings. Furthermore, we propose a joint training approach to effectively integrate both strategies, allowing them to reinforce each other for more accurate instruction interpretation and visually coherent edits aligned with user intent. Extensive experiments demonstrate that MIND-Edit outperforms state-of-the-art image editing methods in both quantitative metrics and visual quality, particularly under complex and challenging scenarios.

Figures (8)

• Figure 1: Overview of the proposed MIND-Edit framework. MIND-Edit takes text instructions, original images, and optional editing masks as inputs. It integrates an instruction optimization strategy and an MLLM insight-driven image editing strategy, jointly optimizing instructions and generating visual representations to guide the diffusion model in creating semantically accurate edited images.
• Figure 2: Illustration of the text instruction optimization strategy. A prompt informs the MLLM about the upcoming instruction optimization task. Given an image and an instruction from the user, the MLLM refines the instruction by resolving ambiguities based on visual and textual context.
• Figure 3: Qualitative comparisons on the HumanEdit datasetbai2024humanedit. A mask is provided for each sample. MIND-Edit achieves superior instruction alignment and visual quality, surpassing or matching other methods even though MagicQuill's generation branch alone contains twice the parameters of our method.
• Figure 4: Qualitative comparisons on the ComplexMultistepImageEditing datasetcomplex-multistep-image-editing-dataset, where no mask is provided. MIND-Edit achieves precise semantic alignment in complex editing scenarios.
• Figure 5: Qualitative results of the ablation study. With the proposed instruction optimization, visual representation generation strategies, and joint training approach, MIND-Edit achieves improved instruction-aligned details, textures, and overall visual consistency compared to other variants.