Tuning-Free Visual Customization via View Iterative Self-Attention Control

TL;DR

VisCtrl introduces a training-free approach for personalized visual editing by injecting a user-specified subject from a reference image into a target image using iterative self-attention within a latent diffusion model. The method uses DDIM inversion to obtain initial noises for both reference and target, and during denoising, replaces the target's self-attention keys/values with those from the reference to progressively fuse appearance and structure, controlled by start step $S$ and layer $L$. A Feature Gradually Sampling strategy extends the method to multi-view domains, enabling consistent editing in videos and 3D scenes by randomly sampling latent features from multiple references and weighting updates with $\alpha$. Extensive experiments across images, video, and 3D scenes show VisCtrl outperforms exemplar-based baselines in fidelity and layout preservation, while remaining training-free and plug-and-play.

Abstract

Fine-Tuning Diffusion Models enable a wide range of personalized generation and editing applications on diverse visual modalities. While Low-Rank Adaptation (LoRA) accelerates the fine-tuning process, it still requires multiple reference images and time-consuming training, which constrains its scalability for large-scale and real-time applications. In this paper, we propose \textit{View Iterative Self-Attention Control (VisCtrl)} to tackle this challenge. Specifically, VisCtrl is a training-free method that injects the appearance and structure of a user-specified subject into another subject in the target image, unlike previous approaches that require fine-tuning the model. Initially, we obtain the initial noise for both the reference and target images through DDIM inversion. Then, during the denoising phase, features from the reference image are injected into the target image via the self-attention mechanism. Notably, by iteratively performing this feature injection process, we ensure that the reference image features are gradually integrated into the target image. This approach results in consistent and harmonious editing with only one reference image in a few denoising steps. Moreover, benefiting from our plug-and-play architecture design and the proposed Feature Gradual Sampling strategy for multi-view editing, our method can be easily extended to edit in complex visual domains. Extensive experiments show the efficacy of VisCtrl across a spectrum of tasks, including personalized editing of images, videos, and 3D scenes.

Figures (13)

• Figure 1: VisCtrl results span across various object and image domains, showcasing its broad applicability. From simple objects (cartoons, logos) to complex subjects (food, humans), the diversity in personalized image editing highlights the versatility and robustness of our framework across different usage scenarios.
• Figure 2: Pipeline of the proposed VisCtrl. Given one or several reference images of a new concept, we first encoder them to the latent space, followed by adding noise and denoising via DDIM song2021ddim. The upper part of the process entails generating the reference image, while the bottom part involves generating the target image. Specifically, during the denoising process, we replace the $K_t, V_t$ of the target image self-attention layer with $K_s, V_s$ from the reference image self-attention layer. Additionally, we update $Z^*$ with $Z_0^t$ iteratively throughout this process. Finally, we decoder $Z_0^t$ to obtain the target image. Please refer to Section \ref{['sec:visctrl']} for further details.
• Figure 3: Cross-Attention maps under different iterations. On the left, using the VisCtrl method, the appearance of a reference image with text condition $\mathcal{P}_s$ is inserted into a target image with text condition $\mathcal{P}_t$. On the right are the changes in the target image during the iterations, as well as the changes in the cross-attention computed between its intermediate latent and $\mathcal{P}_s$ and $\mathcal{P}_t$ respectively. Please refer to Section \ref{['sec:2d_editing']} for more details.
• Figure 4: Results of different methods on personalized image editing. Our proposed VisCtrl method yields compelling results across various object and image domains, showcasing its broad applicability. From left to right: the reference image and the source image with their respective prompts, editing results with the proposed VisCtrl method, and Other Exemplar-guided Image Editing results with existing methods AnyDoor chen2023anydoor, Paint by Example yang2022paint, and Photoswap gu2023photoswap. Please refer to Section \ref{['sec:comparison']} for more details.
• Figure 5: Results of different methods on personalized video editing. We edit the foreground subject and background of various videos using different methods. Compare to baseline, Our method not only generates content that is more similar to the reference image but also maintains the continuity of the edited regions across different frames.