Shape-Guided Diffusion with Inside-Outside Attention

TL;DR

Shape-Guided Diffusion tackles the challenge of preserving precise object silhouettes during text-guided edits by introducing Inside-Outside Attention, a training-free mechanism that constrains cross- and self-attention maps to distinguish inside (object) from outside (background) regions during both inversion and generation. The approach builds on a frozen latent diffusion model and can use provided masks or inferred shapes from prompts, achieving state-of-the-art shape faithfulness on the MS-COCO ShapePrompts benchmark while maintaining text alignment and image realism. Quantitative metrics ($mIoU$, $KW\text{-}mIoU$, $FID$, $CLIP$) and annotator ratings confirm substantial improvements over prior methods, with the added ability to perform intra-/inter-class, inside/outside, and background edits via a training-free, inference-time procedure. The work also introduces a dedicated ShapePrompts benchmark, provides extensive ablations, and demonstrates robustness to inferred masks, offering a practical tool for shape-aware diffusion-based editing in real-world workflows.

Abstract

We introduce precise object silhouette as a new form of user control in text-to-image diffusion models, which we dub Shape-Guided Diffusion. Our training-free method uses an Inside-Outside Attention mechanism during the inversion and generation process to apply a shape constraint to the cross- and self-attention maps. Our mechanism designates which spatial region is the object (inside) vs. background (outside) then associates edits to the correct region. We demonstrate the efficacy of our method on the shape-guided editing task, where the model must replace an object according to a text prompt and object mask. We curate a new ShapePrompts benchmark derived from MS-COCO and achieve SOTA results in shape faithfulness without a degradation in text alignment or image realism according to both automatic metrics and annotator ratings. Our data and code will be made available at https://shape-guided-diffusion.github.io.

Figures (22)

• Figure 1: We demonstrate that prior work in local image editing hertz2022promptmeng2022sdeditsdinpaintavrahami2022blended fails to preserve precise object silhouette. We propose Shape-Guided Diffusion, a training-free method that uses a novel Inside-Outside Attention to respect shape input. Our method can be provided an object mask as input or infer a mask from text.
• Figure 2: Our work differs from concurrent work in structure-preserving editing mokady2022nulltexttumanyan2022pnpbrooks2022instructpix2pix in that we constrain attention maps such that edits are localized to a spatial region. Here we infer our shape constraint from the text prompt, thereby using the same amount of input as other methods.
• Figure 3: Shape-Guided Diffusion. Our method takes a real image, source prompt ("dog"), edit prompt ("dog wearing a colorful shirt"), as well as an optional object mask, and outputs an edited image. We infer the object mask from the source prompt if it is not provided using a shape inference function, e.g., a segmentation model. Left: we modify a frozen pretrained text-to-image diffusion model during both the inversion and generation processes. Right: we show a detailed view of one layer in the U-Net, where Inside-Outside Attention constrains the self- and cross-attention maps according to the mask.
• Figure 4: Inside-Outside Attention. We modify both the cross- and self-attention maps. Here $j$ refers to token/pixel indices and $M_{*j}$ denotes the attention map corresponding to the $j$-th index. Cross-Attn Layer (top): depending on whether the text embedding refers to the inside or outside the object, we constrain the attention map $M$ according to the object mask or the inverted object mask to produce $M'$. Self-Attn Layer (bottom): we perform a similar operation on the inside and outside pixel embeddings.
• Figure 5: Spurious attentions and classifier-free guidance limits shape preservation. Inside-Outside Attention (top) preserves the shape relationship between the object and background by associating tokens to specific spatial regions. We demonstrate this property when reconstructing (left) and editing (right) a real image with classifier-free guidance. We also depict the cross attention map for the token "dog" averaged all attention heads and timesteps.