Region in Context: Text-condition Image editing with Human-like semantic reasoning

TL;DR

Region in Context tackles the challenge of maintaining global semantic coherence when editing image regions conditioned on text. The authors introduce a dual-level framework that combines region-level alignment with local region descriptions and a global scene-guided alignment using scene-level descriptions generated by a vision-language model, reinforced by gated region-context fusion. The approach leverages CLIP for region embeddings and BLIP for scene embeddings, plus DeepSeek-VL for dense LLM-provided scene references, trained with a region-focused contrastive loss and a global alignment loss on the HumanEdit dataset. Empirical results show consistently improved semantic fidelity and perceptual quality across multiple baselines, highlighting the importance of holistic context in instruction-driven image editing. Code is available at https://github.com/thuyvuphuong/Region-in-Context.git.

Abstract

Recent research has made significant progress in localizing and editing image regions based on text. However, most approaches treat these regions in isolation, relying solely on local cues without accounting for how each part contributes to the overall visual and semantic composition. This often results in inconsistent edits, unnatural transitions, or loss of coherence across the image. In this work, we propose Region in Context, a novel framework for text-conditioned image editing that performs multilevel semantic alignment between vision and language, inspired by the human ability to reason about edits in relation to the whole scene. Our method encourages each region to understand its role within the global image context, enabling precise and harmonized changes. At its core, the framework introduces a dual-level guidance mechanism: regions are represented with full-image context and aligned with detailed region-level descriptions, while the entire image is simultaneously matched to a comprehensive scene-level description generated by a large vision-language model. These descriptions serve as explicit verbal references of the intended content, guiding both local modifications and global structure. Experiments show that it produces more coherent and instruction-aligned results. Code is available at: https://github.com/thuyvuphuong/Region-in-Context.git

Figures (14)

• Figure 1: Illustration of our vision-language embedding optimization strategy. At the global level (left), the model aligns image and full target description embeddings. At the local level (right), it aligns the region in the full image to make the region "see itself in context", with their corresponding region descriptions using contrastive learning.
• Figure 2: Effectiveness of the proposed training framework through comparing models finetuned with and without our framework.
• Figure 3: The proposed region-aware diffusion framework for text-guided image editing. The model aligns visual and textual representations at both region and scene levels using a gated region-context fusion module and contrastive supervision. A denoising process reconstructs the edited image from noisy input while ensuring semantic fidelity through region-level and global alignment losses.
• Figure 4: Gated Cross-Attention Fusion module. Region and full image embeddings are first transformed, then fused via multi-head cross-attention. A learnable gate regulates the influence of global context on the region, ensuring controlled and context-aware representation refinement.
• Figure 5: Qualitative comparison with baseline methods. Our framework produces edits that are both semantically accurate and visually coherent across various instructions. Our results consistently exhibit better blending and alignment with both regional and global semantics.