Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Skip-and-Play: Depth-Driven Pose-Preserved Image Generation for Any Objects

Kyungmin Jo, Jaegul Choo

TL;DR

This work proposes Skip-and-Play (SnP), designed via analysis of the impact of three components of depth-conditional Control-Net on the pose and the shape of the generated images, which exhibits the ability to generate images even when the objects in the condition differ from each other.

Abstract

The emergence of diffusion models has enabled the generation of diverse high-quality images solely from text, prompting subsequent efforts to enhance the controllability of these models. Despite the improvement in controllability, pose control remains limited to specific objects (e.g., humans) or poses (e.g., frontal view) due to the fact that pose is generally controlled via camera parameters (e.g., rotation angle) or keypoints (e.g., eyes, nose). Specifically, camera parameters-conditional pose control models generate unrealistic images depending on the object, owing to the small size of 3D datasets for training. Also, keypoint-based approaches encounter challenges in acquiring reliable keypoints for various objects (e.g., church) or poses (e.g., back view). To address these limitations, we propose depth-based pose control, as depth maps are easily obtainable from a single depth estimation model regardless of objects and poses, unlike camera parameters and keypoints. However, depth-based pose control confronts issues of shape dependency, as depth maps influence not only the pose but also the shape of the generated images. To tackle this issue, we propose Skip-and-Play (SnP), designed via analysis of the impact of three components of depth-conditional ControlNet on the pose and the shape of the generated images. To be specific, based on the analysis, we selectively skip parts of the components to mitigate shape dependency on the depth map while preserving the pose. Through various experiments, we demonstrate the superiority of SnP over baselines and showcase the ability of SnP to generate images of diverse objects and poses. Remarkably, SnP exhibits the ability to generate images even when the objects in the condition (e.g., a horse) and the prompt (e.g., a hedgehog) differ from each other.

Skip-and-Play: Depth-Driven Pose-Preserved Image Generation for Any Objects

TL;DR

This work proposes Skip-and-Play (SnP), designed via analysis of the impact of three components of depth-conditional Control-Net on the pose and the shape of the generated images, which exhibits the ability to generate images even when the objects in the condition differ from each other.

Abstract

The emergence of diffusion models has enabled the generation of diverse high-quality images solely from text, prompting subsequent efforts to enhance the controllability of these models. Despite the improvement in controllability, pose control remains limited to specific objects (e.g., humans) or poses (e.g., frontal view) due to the fact that pose is generally controlled via camera parameters (e.g., rotation angle) or keypoints (e.g., eyes, nose). Specifically, camera parameters-conditional pose control models generate unrealistic images depending on the object, owing to the small size of 3D datasets for training. Also, keypoint-based approaches encounter challenges in acquiring reliable keypoints for various objects (e.g., church) or poses (e.g., back view). To address these limitations, we propose depth-based pose control, as depth maps are easily obtainable from a single depth estimation model regardless of objects and poses, unlike camera parameters and keypoints. However, depth-based pose control confronts issues of shape dependency, as depth maps influence not only the pose but also the shape of the generated images. To tackle this issue, we propose Skip-and-Play (SnP), designed via analysis of the impact of three components of depth-conditional ControlNet on the pose and the shape of the generated images. To be specific, based on the analysis, we selectively skip parts of the components to mitigate shape dependency on the depth map while preserving the pose. Through various experiments, we demonstrate the superiority of SnP over baselines and showcase the ability of SnP to generate images of diverse objects and poses. Remarkably, SnP exhibits the ability to generate images even when the objects in the condition (e.g., a horse) and the prompt (e.g., a hedgehog) differ from each other.
Paper Structure (18 sections, 6 equations, 13 figures)

This paper contains 18 sections, 6 equations, 13 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary
  4. Method
  5. Analysis of ControlNet on the Pose of Image
  6. Time Steps using ControlNet.
  7. ControlNet Features Obtained from Negative Prompt.
  8. ControlNet Features for Each Decoder Block.
  9. Skip-and-Play
  10. Experimetal Results
  11. Comparison of Direct Pose Control
  12. Quantitative Comparison.
  13. Qualitative Comparison.
  14. Comparison to Structure-based Pose Control
  15. Effects on the Shape of Generated Images
  16. ...and 3 more sections

Figures (13)

  • Figure 1: Our method, Skip-and-Play (SnP), generates images of any objects from either image prompts (top) or text prompts (bottom), reflecting the given poses of conditions. While a depth (DP)-conditional ControlNet generates images reflecting object shapes from the condition, SnP produces images where the shapes reflect the prompt rather than the condition, despite employing the same model without additional training. For instance, when using the prompt "pig" and the depth map of a horse image as the condition, ControlNet produces a pig with the shape of a horse, while SnP does not. Extra results and the full text prompts are in the Supplementary (Suppl.).
  • Figure 2: Impact of three components of ControlNet on the pose of the generated images. (a) Impact of NP in the ControlNet encoder. With NP, using ControlNet up to 0.4 time steps leads to a notable decrease in pose error between the generated images and conditions, but using it beyond this step yields marginal improvement in pose error. However, not using NP aids in reflecting the given pose across time steps using ControlNet. ts indicates $\lambda_{t}$. (b) Among the ControlNet features, features for the middle block (MB) and the fourth DB have the most significant impact on the pose.
  • Figure 3: Visual results according to the time steps $\lambda_{t}$ using ControlNet in the blue line in \ref{['fig:skip_neg']}. Using ControlNet up to 0.4 time steps reflects the pose of a given condition, but the shape of the condition is also reflected in the generated image.
  • Figure 4: Visual results of predicted denoised images at each time step with (top) and without (bottom) using ControlNet features from NP. These images depict the visual outcomes of ts0.2 in \ref{['fig:skip_neg']}. Utilizing ControlNet features from NP causes a change in the pose of the image at the moment of cessation of ControlNet usage (blue dashed line) and creates shapes dependent on the depth map when using ControlNet.
  • Figure 5: Generated images using ControlNet features in each decoder block (DB) at a time (Top: without ControlNet features in the middle block (MB), Bottom: with the features in the MB). These correspond to the blue and orange lines in \ref{['fig:skip_block']}, respectively. ControlNet features added to the MB control coarse pose, while those added to the features for the fourth DB adjust fine pose and image shape.
  • ...and 8 more figures