DeepInv: A Novel Self-supervised Learning Approach for Fast and Accurate Diffusion Inversion

TL;DR

DeepInv tackles the lack of supervision in diffusion inversion by learning a trainable, stepwise inversion solver in a self-supervised framework. It builds pseudo labels via a fixed-point, denoising-consistent objective and fuses them with forward-denoising signals in a multi-scale, iterative training regime. A dual-branch solver architecture leverages pretrained priors and image-conditioned refinement, enabling high-fidelity inversions with orders-of-magnitude speedups over prior methods and improving downstream editing when integrated into existing pipelines. The approach demonstrates substantial gains on COCO and PIE-Bench datasets and provides a practical, open-source path toward fast, controllable diffusion-based image editing.

Abstract

Diffusion inversion is a task of recovering the noise of an image in a diffusion model, which is vital for controllable diffusion image editing. At present, diffusion inversion still remains a challenging task due to the lack of viable supervision signals. Thus, most existing methods resort to approximation-based solutions, which however are often at the cost of performance or efficiency. To remedy these shortcomings, we propose a novel self-supervised diffusion inversion approach in this paper, termed Deep Inversion (DeepInv). Instead of requiring ground-truth noise annotations, we introduce a self-supervised objective as well as a data augmentation strategy to generate high-quality pseudo noises from real images without manual intervention. Based on these two innovative designs, DeepInv is also equipped with an iterative and multi-scale training regime to train a parameterized inversion solver, thereby achieving the fast and accurate image-to-noise mapping. To the best of our knowledge, this is the first attempt of presenting a trainable solver to predict inversion noise step by step. The extensive experiments show that our DeepInv can achieve much better performance and inference speed than the compared methods, e.g., +40.435% SSIM than EasyInv and +9887.5% speed than ReNoise on COCO dataset. Moreover, our careful designs of trainable solvers can also provide insights to the community. Codes and model parameters will be released in https://github.com/potato-kitty/DeepInv.

Figures (5)

• Figure 1: Illustrations of our DeepInv and previous inversion strategies. (a) Iterative optimization methods pan2023effectivegaribi2024renoise update the inversion noise at each timestep, but requiring excessive time. (b) Ordinary differential equation (ODE) based approaches rf-invrf-solver design invertible sampler with better efficiency, but are often at the cost of reconstruction quality. (c) DeepInv is the first approach to train an step-by-step inversion solver to fast and accurately predict inversion noise.
• Figure 2: The comparison between our DeepInv and existing SOTA methods, i.e., EasyInv EasyInv, FTEdit FTEdit and DVRF DVRF, in terms of image inversion (a) and editing (b). (a) shows the visualization of image inversions, which shows that our DeepInv can better preserve the details of the original images than EasyInv, e.g., the textures and structure. (b) shows the image editing results according to the text prompt. The inverted noise predict by DeepInv can help FTEdit achieve more better editing results well aligned to the text prompt, while the compared methods are easy to fail in alignments.
• Figure 3: Overview of our proposed training framework. We begin by initializing a base network of the inversion solver. The training process proceeds through 4 iterative stages. In each iteration, the solver is trained under 5 different timestep configurations, and for each configuration, 2 rounds of optimization are performed using distinct loss functions. During the 3 iteration, additional layers are appended and trained while the previously learned parameters are frozen. In the 4 iteration (the last one), all parameters are jointly fine-tuned with a learning rate reduced to 10% of the original. Further implementation details are provided in Section Method.
• Figure 4: Architecture of the proposed DeepInv solver. The dual-branch design processes noise (left) and image (right) information separately before final aggregation. Extra blocks added in second last round to extend model ability.
• Figure 5: Visualized comparison between DeepInv Solver and existing methods on the task of image inversion (left) and editing (right), respectively. For the image editing task, we integrate DeepInv solver into two representative inversion-based diffusion editing methods, i.e., RF-Inv rf-inv and FTEdit FTEdit, by replacing their original inversion modules. DVRF DVRF is a SOTA and inversion-free method. In each example, the object in the original image is marked in red, and the replaced (or added) object is highlighted in blue. The first-row example illustrates an object-addition scenario with only the blue prompts. According to shown images, DeepInv solver consistently achieves more faithful inversions and leads to visually coherent edits.