Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gradient-free Decoder Inversion in Latent Diffusion Models

Seongmin Hong, Suh Yoon Jeon, Kyeonghyun Lee, Ernest K. Ryu, Se Young Chun

TL;DR

The paper tackles decoder inversion in latent diffusion models by proposing a gradient-free approach that replaces gradient backpropagation with a forward-step update guided by the encoder and decoder. It provides theoretical convergence guarantees under $\beta$-cocoercivity for the vanilla forward step and for inertial Krasnoselskii-Mann momentum, and validates these assumptions empirically across multiple LDMs including SD 2.1, LaVie, and InstaFlow. Practical enhancements using the Adam optimizer and learning-rate scheduling yield substantial reductions in runtime and peak memory, making 16-bit inference feasible and enabling efficient inversion for video-scale latent spaces. The gradient-free method is applied to tree-rings watermarking to achieve accurate classification with lower resource demands, illustrating concrete benefits for invertibility tasks and potential broader impacts in content protection and editing workflows.

Abstract

In latent diffusion models (LDMs), denoising diffusion process efficiently takes place on latent space whose dimension is lower than that of pixel space. Decoder is typically used to transform the representation in latent space to that in pixel space. While a decoder is assumed to have an encoder as an accurate inverse, exact encoder-decoder pair rarely exists in practice even though applications often require precise inversion of decoder. Prior works for decoder inversion in LDMs employed gradient descent inspired by inversions of generative adversarial networks. However, gradient-based methods require larger GPU memory and longer computation time for larger latent space. For example, recent video LDMs can generate more than 16 frames, but GPUs with 24 GB memory can only perform gradient-based decoder inversion for 4 frames. Here, we propose an efficient gradient-free decoder inversion for LDMs, which can be applied to diverse latent models. Theoretical convergence property of our proposed inversion has been investigated not only for the forward step method, but also for the inertial Krasnoselskii-Mann (KM) iterations under mild assumption on cocoercivity that is satisfied by recent LDMs. Our proposed gradient-free method with Adam optimizer and learning rate scheduling significantly reduced computation time and memory usage over prior gradient-based methods and enabled efficient computation in applications such as noise-space watermarking while achieving comparable error levels.

Gradient-free Decoder Inversion in Latent Diffusion Models

TL;DR

The paper tackles decoder inversion in latent diffusion models by proposing a gradient-free approach that replaces gradient backpropagation with a forward-step update guided by the encoder and decoder. It provides theoretical convergence guarantees under -cocoercivity for the vanilla forward step and for inertial Krasnoselskii-Mann momentum, and validates these assumptions empirically across multiple LDMs including SD 2.1, LaVie, and InstaFlow. Practical enhancements using the Adam optimizer and learning-rate scheduling yield substantial reductions in runtime and peak memory, making 16-bit inference feasible and enabling efficient inversion for video-scale latent spaces. The gradient-free method is applied to tree-rings watermarking to achieve accurate classification with lower resource demands, illustrating concrete benefits for invertibility tasks and potential broader impacts in content protection and editing workflows.

Abstract

In latent diffusion models (LDMs), denoising diffusion process efficiently takes place on latent space whose dimension is lower than that of pixel space. Decoder is typically used to transform the representation in latent space to that in pixel space. While a decoder is assumed to have an encoder as an accurate inverse, exact encoder-decoder pair rarely exists in practice even though applications often require precise inversion of decoder. Prior works for decoder inversion in LDMs employed gradient descent inspired by inversions of generative adversarial networks. However, gradient-based methods require larger GPU memory and longer computation time for larger latent space. For example, recent video LDMs can generate more than 16 frames, but GPUs with 24 GB memory can only perform gradient-based decoder inversion for 4 frames. Here, we propose an efficient gradient-free decoder inversion for LDMs, which can be applied to diverse latent models. Theoretical convergence property of our proposed inversion has been investigated not only for the forward step method, but also for the inertial Krasnoselskii-Mann (KM) iterations under mild assumption on cocoercivity that is satisfied by recent LDMs. Our proposed gradient-free method with Adam optimizer and learning rate scheduling significantly reduced computation time and memory usage over prior gradient-based methods and enabled efficient computation in applications such as noise-space watermarking while achieving comparable error levels.
Paper Structure (41 sections, 3 theorems, 37 equations, 8 figures, 5 tables)

This paper contains 41 sections, 3 theorems, 37 equations, 8 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Backgrounds
  3. Latent diffusion models (LDMs)
  4. Exact inversion of accelerated DMs.
  5. Optimization-based GAN inversion
  6. Gradient-based decoder inversion in LDMs
  7. Gradient-free decoder inversion in LDMs
  8. Motivation
  9. Convergence analysis on forward step method
  10. Convergence analysis on momentum for acceleration
  11. Validation of the assumption
  12. Experiments with practical optimization techniques
  13. Adam optimizer.
  14. (Optional) Learning rate scheduling.
  15. Results.
  16. ...and 26 more sections

Key Result

Theorem 1

Let $\beta>0$, $0<\rho<2\beta$, and ${\bm{x}} \in \mathbb{R}^N$. Assume ${\mathcal{T}}(\cdot)={\mathcal{E}} \circ {\mathcal{D}}(\cdot) - {\mathcal{E}}({\bm{x}})$ is continuous. Consider the iteration Assume ${\bm{z}}^\star$ is a zero of ${\mathcal{T}}$ (i.e., ${\mathcal{T}} {\bm{z}}^\star = 0$) and Then, ${\mathcal{T}} {\bm{z}}^k \rightarrow 0$. If, furthermore, ${\bm{z}}^k \rightarrow {\bm{z}}^

Figures (8)

  • Figure 1: (a) The difference between LDM and pixel-space DM lies in the use of the decoder (${\mathcal{D}}$). (b) This difference has caused the performance gap in exact inversions. Pixel-space DMs and gradient-based GAN inversion methods are located on the right side due to iterative gradient back-propagations. If gradient-based decoder inversions are used, which are computationally intensive, LDM is located on the rightmost side to address the lossiness of latents. Our proposed gradient-free decoder inversion method allows us to efficiently handle the transformation between latent and pixel spaces.
  • Figure 2: The six subfigures represent the relationship between cocoercivity and convergence for 3 models $\times$ 2 algorithms (vanilla forward step method and inertial KM iteration). The x-axis represents the values of $\underset{k\in[0,100]}{\min}\frac{\langle {\mathcal{E}} {\mathcal{D}} {\bm{z}}^{\infty} - {\mathcal{E}} {\mathcal{D}} {\bm{z}}^{k}, {\bm{z}}^{\infty} - {\bm{z}}^{k} \rangle }{ \lVert {\mathcal{E}} {\mathcal{D}} {\bm{z}}^{\infty} - {\mathcal{E}} {\mathcal{D}} {\bm{z}}^{k} \rVert_2^2}$, which informs whether the optimization path satisfies the assumptions of \ref{['thm:1', 'thm:2']}, while the y-axis represents the convergence (i.e., $\lVert {\bm{z}}^{100} - {\bm{z}}^\infty \rVert_2$). The red line shows the linear function fitted by least squares. We set ${\bm{z}}^{\infty}={\bm{z}}^{300}$.
  • Figure 3: Our gradient-free decoder inversion has a way shorter runtime than the gradient-based decoder inversion, and drastically reduces the GPU memory usage, on (a) SD2.1 rombach2022high, (b) LaVie wang2023lavie, and (c) InstaFlow liu2023instaflow. Note that 16-bit gradient-based approach is unimplementable, due to the underflow problem. Each point represents a different hyperparameter setting (e.g., the total number of iterations, learning rate, learning rate scheduling); by collecting experimental results from such diverse settings, the Pareto frontier can be obtained fairly without manipulation.
  • Figure 4: The assumptions also hold for Adam in many cases, as similar in \ref{['fig:cocoercivity']}. The x-axis represents the values of $\underset{k\in[0,50]}{\min}\frac{\langle {\mathcal{E}} {\mathcal{D}} {\bm{z}}^{\infty} - {\mathcal{E}} {\mathcal{D}} {\bm{z}}^{k}, {\bm{z}}^{\infty} - {\bm{z}}^{k} \rangle }{ \lVert {\mathcal{E}} {\mathcal{D}} {\bm{z}}^{\infty} - {\mathcal{E}} {\mathcal{D}} {\bm{z}}^{k} \rVert_2^2}$, which informs whether the optimization path satisfies the assumptions of \ref{['thm:1', 'thm:2']}, while the y-axis represents the convergence (i.e., $\lVert {\bm{z}}^{50} - {\bm{z}}^\infty \rVert_2$). The red line shows the linear function fitted by least squares. Note that ${\bm{z}}^{\infty}$ was approximated by ${\bm{z}}^{300}$.
  • Figure S1: Confusion matrices for watermark classification on LDMs. Ours is better than the encoder and works as well as the grad-based method hong2023exact.
  • ...and 3 more figures

Theorems & Definitions (10)

  • Remark 1: (\ref{['eqn:find_latent']} is easier than \ref{['eqn:find_pixel']})
  • Theorem 1: Convergence of the forward step method
  • proof : Proof outline
  • Remark 2: $ED={\mathcal{I}}$ in linear autoencoders
  • Theorem 2: Convergence of the inertial KM iterations
  • proof
  • proof : Proof of \ref{['thm:1']}
  • Lemma S1
  • proof : Proof of \ref{['lemma1']}
  • proof : Proof of \ref{['thm:2']}