FlowOpt: Fast Optimization Through Whole Flow Processes for Training-Free Editing

TL;DR

FlowOpt introduces a zero-order, training-free framework that optimizes over the entire flow-based sampling process by treating the denoiser chain as a black box. It provides a provable convergence bound on the step size $\eta$ via a contraction-mapping analysis and empirically estimates this bound for practical models, enabling stable optimization with a fixed budget of NFEs. Empirically, FlowOpt achieves state-of-the-art performance in both image inversion and text-guided editing on high-resolution images, while offering intermediate results for user inspection and early stopping. The approach broadens the utility of pre-trained diffusion/flow models for training-free editing tasks and has potential applications across modalities and tasks such as restoration, compression, and personalization.

Abstract

The remarkable success of diffusion and flow-matching models has ignited a surge of works on adapting them at test time for controlled generation tasks. Examples range from image editing to restoration, compression and personalization. However, due to the iterative nature of the sampling process in those models, it is computationally impractical to use gradient-based optimization to directly control the image generated at the end of the process. As a result, existing methods typically resort to manipulating each timestep separately. Here we introduce FlowOpt - a zero-order (gradient-free) optimization framework that treats the entire flow process as a black box, enabling optimization through the whole sampling path without backpropagation through the model. Our method is both highly efficient and allows users to monitor the intermediate optimization results and perform early stopping if desired. We prove a sufficient condition on FlowOpt's step-size, under which convergence to the global optimum is guaranteed. We further show how to empirically estimate this upper bound so as to choose an appropriate step-size. We demonstrate how FlowOpt can be used for image editing, showcasing two options: (i) inversion (determining the initial noise that generates a given image), and (ii) directly steering the edited image to be similar to the source image while conforming to a target text prompt. In both cases, FlowOpt achieves state-of-the-art results while using roughly the same number of neural function evaluations (NFEs) as existing methods. Code and examples are available on the project's webpage.

Figures (27)

• Figure 1: FlowOpt. We propose a zero-order (gradient-free) framework for optimization through an unrolled flow sampling process. FlowOpt can efficiently optimize losses on the target image, even when working with large models and high resolution images. We leverage our framework for text-based image editing, demonstrating state-of-the-art results on both FLUX (first and third rows) and Stable Diffusion 3 (second row). Fine details are visible upon zooming in.
• Figure 2: A whole flow process as a black box. We encapsulate the flow process as a black box function $f$, which receives an initial noise $\boldsymbol{z}_1$ and text conditioning $c$, and outputs a clean sample $\boldsymbol{z}_0$. Each internal step within the black box is given by $\boldsymbol{\psi}_t(\boldsymbol{z}_t, c) = \boldsymbol{z}_t + \boldsymbol{v}_t(\boldsymbol{z}_t, c)\Delta t$, where $\boldsymbol{v}_t$ is the text-conditioned velocity predicting network.
• Figure 3: Image inversion with FlowOpt. Intermediate samples $\boldsymbol{z}_0^{(i)} = f(\boldsymbol{z}^{(i)}_t, c)$ attained during our zero-order optimization through a chain of $10$ denoising steps (FLUX) for the task of reconstruction (inversion), i.e., with $c=c_{\text{src}}$. Notice the missing details in the early steps, such as the bicycle and the horizon. As the iterations progress, the reconstruction converges to the ground truth image.
• Figure 4: Direct image editing with FlowOpt. Intermediate samples $\boldsymbol{z}_0^{(i)} = f(\boldsymbol{z}^{(i)}_t, c)$ attained during our zero-order optimization through a chain of $15$ denoising steps (FLUX) for direct image editing, i.e., with $c=c_{\text{tar}}$. Notice the misalignment in the dog's body structure in the first iterations.
• Figure 5: Convergence analysis. The plot shows RMSE in pixel space vs. number of iterations for the task of inversion, averaged over a dataset. The step size we use (red) satisfies the sufficient condition of Eq. (\ref{['eq:contraction_mapping_sufficient']}) and thus leads to convergence. Step sizes that are $4\times$ and $5\times$ larger (yellow and black) do not satisfy the condition and do not lead to convergence. The dashed orange line is the minimal RMSE achievable in this setting. It corresponds to passing images through the encoder and decoder.

Theorems & Definitions (1)

• Theorem 1