Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Latent Inversion with Timestep-aware Sampling for Training-free Non-rigid Editing

Yunji Jung, Seokju Lee, Tair Djanibekov, Hyunjung Shim, Jong Chul Ye

TL;DR

This work proposes a training-free approach for non-rigid editing with Stable Diffusion, aimed at improving the identity preservation quality without compromising editability, and introduces latent inversion to preserve the input image's identity without additional model fine-tuning.

Abstract

Text-guided non-rigid editing involves complex edits for input images, such as changing motion or compositions within their surroundings. Since it requires manipulating the input structure, existing methods often struggle with preserving object identity and background, particularly when combined with Stable Diffusion. In this work, we propose a training-free approach for non-rigid editing with Stable Diffusion, aimed at improving the identity preservation quality without compromising editability. Our approach comprises three stages: text optimization, latent inversion, and timestep-aware text injection sampling. Inspired by the success of Imagic, we employ their text optimization for smooth editing. Then, we introduce latent inversion to preserve the input image's identity without additional model fine-tuning. To fully utilize the input reconstruction ability of latent inversion, we suggest timestep-aware text injection sampling. This effectively retains the structure of the input image by injecting the source text prompt in early sampling steps and then transitioning to the target prompt in subsequent sampling steps. This strategic approach seamlessly harmonizes with text optimization, facilitating complex non-rigid edits to the input without losing the original identity. We demonstrate the effectiveness of our method in terms of identity preservation, editability, and aesthetic quality through extensive experiments.

Latent Inversion with Timestep-aware Sampling for Training-free Non-rigid Editing

TL;DR

This work proposes a training-free approach for non-rigid editing with Stable Diffusion, aimed at improving the identity preservation quality without compromising editability, and introduces latent inversion to preserve the input image's identity without additional model fine-tuning.

Abstract

Text-guided non-rigid editing involves complex edits for input images, such as changing motion or compositions within their surroundings. Since it requires manipulating the input structure, existing methods often struggle with preserving object identity and background, particularly when combined with Stable Diffusion. In this work, we propose a training-free approach for non-rigid editing with Stable Diffusion, aimed at improving the identity preservation quality without compromising editability. Our approach comprises three stages: text optimization, latent inversion, and timestep-aware text injection sampling. Inspired by the success of Imagic, we employ their text optimization for smooth editing. Then, we introduce latent inversion to preserve the input image's identity without additional model fine-tuning. To fully utilize the input reconstruction ability of latent inversion, we suggest timestep-aware text injection sampling. This effectively retains the structure of the input image by injecting the source text prompt in early sampling steps and then transitioning to the target prompt in subsequent sampling steps. This strategic approach seamlessly harmonizes with text optimization, facilitating complex non-rigid edits to the input without losing the original identity. We demonstrate the effectiveness of our method in terms of identity preservation, editability, and aesthetic quality through extensive experiments.
Paper Structure (12 sections, 5 equations, 5 figures)

This paper contains 12 sections, 5 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Related Works
  3. Text-guided Image Manipulation
  4. Diffusion Inversion
  5. Preliminary
  6. Method
  7. Experiments
  8. Implementation Details
  9. Qualitative Evaluation
  10. Quantitative Evaluation
  11. Ablation Experiments
  12. Conclusion

Figures (5)

  • Figure 1: Ablation on Imagic and Our method. The images are edited according to the target text: "A photo of a white horse jumping" and "A photo of a couple hugging on a beach".
  • Figure 2: Overview of our method.
  • Figure 3: Qualitative comparison with state-of-the-art methods on Stable Diffusion v1.4.
  • Figure 4: User study and quantitative comparison.
  • Figure 5: Ablation on inversion type.