ST-LDM: A Universal Framework for Text-Grounded Object Generation in Real Images

TL;DR

This work addresses Text-grounded Object Generation (TOG), a task to insert a new object into real images based on natural language while preserving scene semantics. It introduces ST-LDM, a universal framework that adds training-free backward guidance to latent diffusion models through two parallel branches: a Swin-based pixel-space autoencoder for hierarchical features and a deformable multimodal transformer for region-wise spatial guidance. The deformable feature alignment mechanism learns 2D offsets and modulation to steer cross-attention toward linguistically grounded regions, enabling precise region grounding without re-training the diffusion backbone. Quantitative and qualitative evaluations on VrR-VG and ComCOCO-based benchmarks show improved localization of generated content and maintained generative fidelity, with ablations validating the contribution of DFA and backward guidance. Overall, ST-LDM provides a scalable, language-driven approach to inserting new objects into real images and can be integrated with various pretrained latent diffusion models.

Abstract

We present a novel image editing scenario termed Text-grounded Object Generation (TOG), defined as generating a new object in the real image spatially conditioned by textual descriptions. Existing diffusion models exhibit limitations of spatial perception in complex real-world scenes, relying on additional modalities to enforce constraints, and TOG imposes heightened challenges on scene comprehension under the weak supervision of linguistic information. We propose a universal framework ST-LDM based on Swin-Transformer, which can be integrated into any latent diffusion model with training-free backward guidance. ST-LDM encompasses a global-perceptual autoencoder with adaptable compression scales and hierarchical visual features, parallel with deformable multimodal transformer to generate region-wise guidance for the subsequent denoising process. We transcend the limitation of traditional attention mechanisms that only focus on existing visual features by introducing deformable feature alignment to hierarchically refine spatial positioning fused with multi-scale visual and linguistic information. Extensive Experiments demonstrate that our model enhances the localization of attention mechanisms while preserving the generative capabilities inherent to diffusion models.

Figures (11)

• Figure 1: Text-grounded object generation in real images. Our method can generate the rational object within real-world input images based on textual descriptions encompassing its visual characteristic and spatial attribute. The generated object seamlessly integrates with the original image, exhibiting a visually harmonious presence while preserving the contextual semantics of the original scene (e.g. consistency in shadow (top left), geometry (top right), illumination (bottom left), occlusion (bottom right)).
• Figure 2: The basic framework of the proposed ST-LDM. Given a real image and text descriptions, we initially encode them into visual features $V_i$ and text embedding $L$, where $L$ comprises appearance prompt $L_a$ and spatial conditions $L_s$. We employ the Swin-T structure to hierarchically integrate $L$ into $V_i$, where the multimodal map $M_i$ is deformed by learning multi-scale visual and semantic relationships. The generated spatial guidance attention map $G$ and textual prompt $L_a$ are integrated into latent diffusion models with training-free guidance. The autoencoder-decoder in pixel space and the latent diffusion model in latent space retain original structure and parameters, and these components are frozen throughout the training process.
• Figure 3: An illustration of deformable feature alignment. Global visual feature $V_i$, multimodal feature $M_i$, and text feature $L$ are concatenated and inputted into the Transformer layer to learn offsets and modulation scalar. The multi-head cross attention is then performed to generate the refined attention map with sampled queries in $M_i$ and language keys and values.
• Figure 4: The first column corresponds to the input real image, and we visualize the guidance attention map $G$ and corresponding edited results under different prompts with identical conditions (top) and identical prompts with different conditions (bottom).
• Figure 5: Various results produced by ST-LDM, showcasing diverse appearances while adhering to the provided text descriptions.