Text2Place: Affordance-aware Text Guided Human Placement

TL;DR

This work tackles semantic human placement by learning text-guided affordances to insert a given subject into diverse scenes. It introduces a two-stage SHP pipeline: (i) semantic mask optimization using Score Distillation Sampling with a compact blob-based mask to locate plausible placement regions, and (ii) subject-conditioned inpainting using Textual Inversion to preserve identity and adapt pose to the scene. The method leverages rich priors from text-to-image diffusion models to avoid large-scale training and demonstrates realistic placements, scene/human hallucination, and text-based editing across indoor and outdoor scenes. It discusses limitations and ethical considerations while showing competitive results against strong baselines on a newly assembled in-the-wild dataset.

Abstract

For a given scene, humans can easily reason for the locations and pose to place objects. Designing a computational model to reason about these affordances poses a significant challenge, mirroring the intuitive reasoning abilities of humans. This work tackles the problem of realistic human insertion in a given background scene termed as \textbf{Semantic Human Placement}. This task is extremely challenging given the diverse backgrounds, scale, and pose of the generated person and, finally, the identity preservation of the person. We divide the problem into the following two stages \textbf{i)} learning \textit{semantic masks} using text guidance for localizing regions in the image to place humans and \textbf{ii)} subject-conditioned inpainting to place a given subject adhering to the scene affordance within the \textit{semantic masks}. For learning semantic masks, we leverage rich object-scene priors learned from the text-to-image generative models and optimize a novel parameterization of the semantic mask, eliminating the need for large-scale training. To the best of our knowledge, we are the first ones to provide an effective solution for realistic human placements in diverse real-world scenes. The proposed method can generate highly realistic scene compositions while preserving the background and subject identity. Further, we present results for several downstream tasks - scene hallucination from a single or multiple generated persons and text-based attribute editing. With extensive comparisons against strong baselines, we show the superiority of our method in realistic human placement.

Figures (22)

• Figure 1: (Top): Proposed approach for text-based placement of humans. Given a background image, we predict the plausible semantic region compatible with the text prompt to place humans. Next, given a few subject images, we perform subject-conditioned inpainting to realistically place humans in appropriate poses following the scene affordances. (Bottom): Our method enables a) realistic human placements at diverse locations and poses and several downstream applications b) scene hallucination by generating compatible scenes for the given pose of the human c) text-based editing of the human and d) placing multiple persons.
• Figure 2: Our approach consists of two stages: a) Semantic Mask Optimization. Given a background image $\mathcal{I}_b$, we initialize a blob mask $\mathcal{M}$ parameterized as Gaussian blobs and a foreground person image $\mathcal{I}_p$. These two images are combined to form a composite image $\mathcal{I}_c$, which is used to compute SDS loss with the action prompt. During optimization, only $\mathcal{M}$ and $\mathcal{I}_p$ are getting updated via $\mathcal{I}_c$. After training $\mathcal{M}$ converge to a plausible human placement region, which is then used for inpainting. b) Subject conditioned inpainting. Given a few subject images, we perform Textual Inversion to obtain its token embedding $\mathbf{V*}$. Next, we use the inpainting pipeline of T2I models to perform personalized inpainting of the subject.
• Figure 3: Training progression, of semantic mask $\mathcal{M}$ and person image $\mathcal{I}_p$ with SDS loss with prompt 'A person sitting on bed'. $\mathcal{M}$ starts to converge at the appropriate person region following prompt.
• Figure 4: Blob mask parameterization.
• Figure 5: Pixel-perfect mask is not effective for T2I inpainting.