InstructGIE: Towards Generalizable Image Editing

TL;DR

InstructGIE tackles the limited generalization of diffusion-based image editing by integrating a VMamba-based visual prompting module, an editing-shift learning mechanism, selective area matching, and language instruction unification within a Stable Diffusion framework enhanced by ControlNet. It also provides the first open dataset for image editing with visual prompts and editing instructions to evaluate in-context capabilities. Through extensive experiments and ablations, the approach demonstrates superior in-context generation and robust generalization to unseen tasks, with substantial improvements in both visual fidelity and adherence to editing prompts. The work offers practical impact by enabling more reliable and detailed edits across diverse domains without extensive per-task fine-tuning, supported by a publicly available dataset for continued research.

Abstract

Recent advances in image editing have been driven by the development of denoising diffusion models, marking a significant leap forward in this field. Despite these advances, the generalization capabilities of recent image editing approaches remain constrained. In response to this challenge, our study introduces a novel image editing framework with enhanced generalization robustness by boosting in-context learning capability and unifying language instruction. This framework incorporates a module specifically optimized for image editing tasks, leveraging the VMamba Block and an editing-shift matching strategy to augment in-context learning. Furthermore, we unveil a selective area-matching technique specifically engineered to address and rectify corrupted details in generated images, such as human facial features, to further improve the quality. Another key innovation of our approach is the integration of a language unification technique, which aligns language embeddings with editing semantics to elevate the quality of image editing. Moreover, we compile the first dataset for image editing with visual prompts and editing instructions that could be used to enhance in-context capability. Trained on this dataset, our methodology not only achieves superior synthesis quality for trained tasks, but also demonstrates robust generalization capability across unseen vision tasks through tailored prompts.

Figures (7)

• Figure 1: Demo results of the proposed InstructGIE framework on various image manipulation tasks to both humans and scenes. By our proposed method, our model can generalize to generate the desired output with great detail qualities.
• Figure 2: Overall architecture of InstructGIE. The lower pipeline is for both training and inference processes where the model obtains unified editing instructions outputted by Instruction Unification Module $\mathcal{U}$ and combines with visual prompted input $\textbf{Img}^{\text{VPcon}}$ to pass through Zero-VMamba integrated Stable Diffusion model with ControlNet for output image. The upper pipeline is for training only which compares output image and training ground truth $\textbf{Img}^{\text{train}}$ and computes editing shift loss $\mathcal{L}_{es}$ with Editing Shift Module and selective area matching loss $\mathcal{L}_{sam}$ with Selective Area Matching Module.
• Figure 3: Effective Reception Field (ERF) of ConvNet, ViT, VMamba based model architectures.
• Figure 4: Dataset generation process Our dataset generation consists of two phases. Data Generation is to generate sets of image pairs under one editing caption. Data Processing is randomly pick image pairs under the same editing instruction and concatenate them together as one input for training.
• Figure 5: Qualitative Comparison on our Test Dataset. We conducted experiments on various scenarios, including human, architecture and landscape.