FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Model

TL;DR

FireEdit tackles the problem of instruction-based image editing in complex scenes by introducing region-aware vision-language tokens that ground editing targets more accurately. It combines a region-enhanced VLM with a diffusion model, leveraging the Time-Aware Target Injection (TATI) and Hybrid Visual Cross Attention (HVCA) to align edits with user instructions while preserving non-edited regions. The method uses region proposals from Deformable DETR, region tokens, and learnable [IMG_i] tokens, with LoRA fine-tuning to adapt the VLM and an adapter to connect image features to the LLM latent space. Extensive experiments on Emu Edit and MagicBrush show state-of-the-art performance in instruction adherence and semantic consistency, outperforming IP2P, MagicBrush, HQ-Edit, UltraEdit, MGIE, and SmartEdit. The results demonstrate FireEdit's potential for precise, high-fidelity editing without manual masks, enabling more robust real-world editing workflows.

Abstract

Currently, instruction-based image editing methods have made significant progress by leveraging the powerful cross-modal understanding capabilities of vision language models (VLMs). However, they still face challenges in three key areas: 1) complex scenarios; 2) semantic consistency; and 3) fine-grained editing. To address these issues, we propose FireEdit, an innovative Fine-grained Instruction-based image editing framework that exploits a REgion-aware VLM. FireEdit is designed to accurately comprehend user instructions and ensure effective control over the editing process. Specifically, we enhance the fine-grained visual perception capabilities of the VLM by introducing additional region tokens. Relying solely on the output of the LLM to guide the diffusion model may lead to suboptimal editing results. Therefore, we propose a Time-Aware Target Injection module and a Hybrid Visual Cross Attention module. The former dynamically adjusts the guidance strength at various denoising stages by integrating timestep embeddings with the text embeddings. The latter enhances visual details for image editing, thereby preserving semantic consistency between the edited result and the source image. By combining the VLM enhanced with fine-grained region tokens and the time-dependent diffusion model, FireEdit demonstrates significant advantages in comprehending editing instructions and maintaining high semantic consistency. Extensive experiments indicate that our approach surpasses the state-of-the-art instruction-based image editing methods. Our project is available at https://zjgans.github.io/fireedit.github.io.

Figures (5)

• Figure 1: Our framework leverages a vision language model (VLM) to guide instruction-based image editing. Our primary innovation is the introduction of region tokens, which enable the VLM to accurately identify edited objects or areas in complex scenarios while preserving high-frequency details in unintended regions during image decoding.
• Figure 2: The overall framework of FireEdit. The core of FireEdit is to conduct region-aware fusion of multi-modal tokens to promote VLMs and facilitate fine-grained, localized alignments between editing instructions and images. It also introduces a hybrid visual cross-attention module to better preserve image details and a time-aware target injection module to edit targets adaptively.
• Figure 3: The proposed hybrid visual cross attention module (HVCA, left) and the time-aware target injection module (TATI, right). HVCA exploits hybrid encoders of pre-trained networks to balance between global semantic information (e.g., multi-modal alignment networks like CLIP) and local details. TATI utilizes timestep embeddings to incorporate target information into the denoising process.
• Figure 4: Qualitative comparison. We compare the editing performance of FireEdit with SOTA methods on the Emu Edit test set. Each editing instruction is written below each row of images. Compared with other SOTA methods, our approach is superior in accurately locating the edited objects or regions and preserving the detailed information of the input image.
• Figure 5: Ablation studies for components in our method.