TIE: Revolutionizing Text-based Image Editing for Complex-Prompt Following and High-Fidelity Editing

TL;DR

This work tackles the challenge of editing images from complex text prompts while preserving fidelity to the original. It introduces a CoT-enabled framework that uses GPT-4V-derived reasoning to decompose prompts, localize edit regions, and describe target areas, then fine-tunes a lightweight multimodal LLM (LISA-13B) with CoT-driven data for precise mask generation and inpainting prompts. The diffusion-based editor (Kandinsky-2.2) is guided by these CoT-informed prompts and masks to produce high-fidelity edits, demonstrating superior complex-prompt following and image integrity compared with state-of-the-art baselines. A new pipeline dataset and CoT-based fine-tuning strategy enable cost-effective, accurate text-guided editing, with potential for seamless integration of future inpainting models and improved fidelity in real-world applications.

Abstract

As the field of image generation rapidly advances, traditional diffusion models and those integrated with multimodal large language models (LLMs) still encounter limitations in interpreting complex prompts and preserving image consistency pre and post-editing. To tackle these challenges, we present an innovative image editing framework that employs the robust Chain-of-Thought (CoT) reasoning and localizing capabilities of multimodal LLMs to aid diffusion models in generating more refined images. We first meticulously design a CoT process comprising instruction decomposition, region localization, and detailed description. Subsequently, we fine-tune the LISA model, a lightweight multimodal LLM, using the CoT process of Multimodal LLMs and the mask of the edited image. By providing the diffusion models with knowledge of the generated prompt and image mask, our models generate images with a superior understanding of instructions. Through extensive experiments, our model has demonstrated superior performance in image generation, surpassing existing state-of-the-art models. Notably, our model exhibits an enhanced ability to understand complex prompts and generate corresponding images, while maintaining high fidelity and consistency in images before and after generation.

Figures (7)

• Figure 1: Illustration of our proposed framework as a Chain-of-Thought (CoT) process.
• Figure 2: Illustration of data preparation for CoT fine-tuning.
• Figure 3: Visualization comparison with MLLM SOTA models.
• Figure 4: Visualization comparison with stable diffusion (SD) SOTA models.
• Figure 5: Image editing for complex-prompt.