Training-Free Style and Content Transfer by Leveraging U-Net Skip Connections in Stable Diffusion

TL;DR

The paper investigates the underexplored role of U-Net skip connections in Stable Diffusion and introduces SkipInject, a training-free approach that leverages the $l=4$ and $l=5$ skip paths to control content and style separately. They demonstrate that the residuals from the third encoder block primarily encode spatial structure, enabling content transfer when injected into another image, while style information flows through the main decoding path. SkipInject provides text-guided editing and style transfer with strong content preservation and structure fidelity, achieving state-of-the-art or on-par performance across benchmarks, and it offers modulation techniques to dial editing intensity and background influence. The work highlights the practical significance of skip connections for controllable diffusion-based editing and points to future extensions, such as handling two-image style transfer for broader applications.

Abstract

Recent advances in diffusion models for image generation have led to detailed examinations of several components within the U-Net architecture for image editing. While previous studies have focused on the bottleneck layer (h-space), cross-attention, self-attention, and decoding layers, the overall role of the skip connections of the U-Net itself has not been specifically addressed. We conduct thorough analyses on the role of the skip connections and find that the residual connections passed by the third encoder block carry most of the spatial information of the reconstructed image, splitting the content from the style, passed by the remaining stream in the opposed decoding layer. We show that injecting the representations from this block can be used for text-based editing, precise modifications, and style transfer. We compare our method, SkipInject, to state-of-the-art style transfer and image editing methods and demonstrate that our method obtains the best content alignment and optimal structural preservation tradeoff.

Figures (13)

• Figure 1: SkipInject: our method uses the l=4 and l=5 skip connections of Stable Diffusion to obtain flexible content and style transformations. From a painted “content image” of a bird, the model smoothly modifies the subject to resemble various species (e.g., sparrow, eagle) while retaining the overall scene. A generated image of foxes is transformed into “three white robots” and “three wolves in the snow,”, with coherent and realistic alterations. Furthermore, the styles of the two content images are altered holistically, in aesthetics, subjects, and settings.
• Figure 2: Examples of image editing results on Wild-TI2I and ImageNet-R-TI2I real and generated images.
• Figure 3: Image editing results on generated faces. We show precise transformations ranging from subtle changes, like makeup and hairstyle adjustments, to more global effects, including zombie-like effects. Our model preserves the core identity of each subject, maintaining facial structure.
• Figure 4: Examples of style transfer results on the Artist dataset jiang_artist_2024.
• Figure 5: Visualization of the effect of switching each group of skip connections. We show the result of each skip connection switched on the respective swapped group. We observe that the h-space has an almost imperceptible effect on the final image, contrary to research into the disentanglement of DDPMs. The first group of skip connections closest to the h-space similarly has a limited effect, whereas the most coherent blending occurs in the second group of skip connections. The third group has no coherent effect on the image, generating random distortions, while the fourth performs akin to raw pixel blending.