InstanceGen: Image Generation with Instance-level Instructions

TL;DR

InstanceGen tackles the gap between complex textual prompts and image fidelity by coupling image-based structural initialization with LLM-driven instance-level instructions, all without additional training. The method generates a fine-grained instance layout from the initial diffusion output, assigns per-segment objects and attributes via an LLM, and enforces these assignments through cross-attention losses, attention masking, and composition-preserving regularization during a second diffusion pass. It is evaluated on DrawBench, GenEval, and a new CompoundPrompts benchmark, showing superior counting accuracy and spatial fidelity, especially for prompts with multiple objects and instance-level attributes. The approach advances zero-shot controllability in text-to-image generation and provides a rigorous benchmark for evaluating complex prompts, with practical impact for precise visual synthesis of structured prompts.

Abstract

Despite rapid advancements in the capabilities of generative models, pretrained text-to-image models still struggle in capturing the semantics conveyed by complex prompts that compound multiple objects and instance-level attributes. Consequently, we are witnessing growing interests in integrating additional structural constraints, typically in the form of coarse bounding boxes, to better guide the generation process in such challenging cases. In this work, we take the idea of structural guidance a step further by making the observation that contemporary image generation models can directly provide a plausible fine-grained structural initialization. We propose a technique that couples this image-based structural guidance with LLM-based instance-level instructions, yielding output images that adhere to all parts of the text prompt, including object counts, instance-level attributes, and spatial relations between instances.

Figures (8)

• Figure 1: Method Overview. Given a text prompt (possibly) containing object counts, instance-level attributes, and spatial relations, our approach combines image-based and text-based components (visualized in pink and blue boxes above) for generating an output image (illustrated on the right). We first generate an initial image using a pretrained text-to-image diffusion model. Given the image and attention information from the initial diffusion process, we extract an instance segmentation layout, and assign object and instance-level instructions using an LLM. We then use the layout and instructions, as well as the initial image's latents, to incur losses, mask the attention, and optimize the latent in the assignment conditioned image generation stage which produces the output image. Note that for simplicity attention masking is not displayed in the figure.
• Figure 2: Instance Layout Generation. (a) The initial image generated by a pretrained text-to-image model. (b) The aggregated cross attention for all foreground words, with anchor points marked by green dots. (c) Initial segmentation after discarding segments without anchor points. (d) Final segmentation after adding segments for each unassigned anchor point.
• Figure 3: Instance assignment inputs. The input given to the LLM for instance assignment contains in-context examples, the parsed prompt, and the visual layout of each segment.
• Figure 4: Qualitative results from the CompoundPrompts benchmark. We present results for two unique prompts from the CompoundPrompts benchmark, presenting results for all three tiers (A, B and C) for each prompt.
• Figure 5: Extended qualitative comparison. Additional results for prompts sampled from CompoundPrompts, along with VQA Acc. metric evaluation. For each tier, we consider an image correct ($\color{green}{\checkmark}$) if it receives positive answers from an MLLM (GPT4o) when prompted with all questions below its tier. We compare with Emu dai2023emuenhancingimagegeneration, Flux1-dev flux2023, SDXL podell2023sdxlimprovinglatentdiffusion, Bounded Attention (BA) dahary2025yourself, Attention Refocusing (GLIGEN + AR) phung2024grounded, and Reason Your Layout (RYL) chen2023reason.