Person-In-Situ: Scene-Consistent Human Image Insertion with Occlusion-Aware Pose Control

TL;DR

This work tackles the challenge of inserting a person into a scene with occlusions and explicit pose control. It introduces two diffusion-based pipelines: a two-stage depth-guided method and a direct occlusion-learning method, both using a rendered 3D SMPL model and depth cues to achieve scene-consistent results. The training data are built from video frames with paired poses and depth information, enabling occlusion learning without explicit occlusion masks, and the methods outperform a prior baseline in both qualitative and quantitative evaluations. The approach holds practical value for advertising and entertainment by enabling controllable, occlusion-aware person insertion that preserves the surrounding scene.

Abstract

Compositing human figures into scene images has broad applications in areas such as entertainment and advertising. However, existing methods often cannot handle occlusion of the inserted person by foreground objects and unnaturally place the person in the frontmost layer. Moreover, they offer limited control over the inserted person's pose. To address these challenges, we propose two methods. Both allow explicit pose control via a 3D body model and leverage latent diffusion models to synthesize the person at a contextually appropriate depth, naturally handling occlusions without requiring occlusion masks. The first is a two-stage approach: the model first learns a depth map of the scene with the person through supervised learning, and then synthesizes the person accordingly. The second method learns occlusion implicitly and synthesizes the person directly from input data without explicit depth supervision. Quantitative and qualitative evaluations show that both methods outperform existing approaches by better preserving scene consistency while accurately reflecting occlusions and user-specified poses.

Figures (9)

• Figure 1: We tackle a novel problem of occlusion-aware human image insertion with explicit pose control, which cannot be handled by the state-of-the-art method Affordanceinsertion. Our method can insert a person in a specified pose at an appropriate depth within a scene, without altering the scene's appearance.
• Figure 2: Our two methods for human image composition: (1) a two-stage estimation method, which first estimates an intermediate depth map and then composites the final output; and (2) a direct estimation method, which synthesizes the composited image in a single step.
• Figure 3: Overview of the dataset creation process. Two frames are randomly sampled from a single video: one is used as the reference human image, and the other as the ground-truth image, enabling training with paired images (and relevant data) of the same person in different poses.
• Figure 4: Network architecture of our two-stage estimation method during inference. In the first stage, the model takes as input the scene image $I_{s}$, reference human image $I_\mathit{ref}$, scene depth map $D_{s}$, 3D human model's depth map $D_{p}$, binary mask $M$ defined by the 3D human model's bounding box, and masked scene depth map $M \! * \! D_s$ to predict a depth map $\hat{D}$ of the scene with the person composited. In the second stage, the model uses $I_{s}$, $I_\mathit{ref}$, $D_{s}$, $D_{p}$, and $\hat{D}$ to generate the final composited image.
• Figure 5: Network architecture of the direct estimation method during inference. Our method takes as input the scene image $I_{s}$, reference human image $I_\mathit{ref}$, scene depth map $D_{s}$, 3D human model's depth map $D_{p}$, binary mask $M$ defined by the 3D human model's bounding box, and masked scene image $M\!*\!I_s$. Our method then composites $I_\mathit{ref}$, posed according to $D_{p}$, into $I_{s}$ at an appropriate depth.