FreeTuner: Any Subject in Any Style with Training-free Diffusion

TL;DR

FreeTuner addresses compositional personalization for diffusion models without training by decoupling content and style generation into two stages and leveraging intermediate diffusion features alongside style guidance. The method requires only one image per concept and avoids training encoders or fine-tuning, achieving subject-driven, style-driven, and subject-style combinations. Its two-stage disentanglement and attention-map manipulation enable faithful subject structure preservation while aligning stylistic attributes, yielding state-of-the-art results in various personalization settings. This approach offers practical impact for rapid, training-free personalizations and can be extended to video generation in future work.

Abstract

With the advance of diffusion models, various personalized image generation methods have been proposed. However, almost all existing work only focuses on either subject-driven or style-driven personalization. Meanwhile, state-of-the-art methods face several challenges in realizing compositional personalization, i.e., composing different subject and style concepts, such as concept disentanglement, unified reconstruction paradigm, and insufficient training data. To address these issues, we introduce FreeTuner, a flexible and training-free method for compositional personalization that can generate any user-provided subject in any user-provided style (see Figure 1). Our approach employs a disentanglement strategy that separates the generation process into two stages to effectively mitigate concept entanglement. FreeTuner leverages the intermediate features within the diffusion model for subject concept representation and introduces style guidance to align the synthesized images with the style concept, ensuring the preservation of both the subject's structure and the style's aesthetic features. Extensive experiments have demonstrated the generation ability of FreeTuner across various personalization settings.

Figures (18)

• Figure 1: Given a subject image and a style image, our training-free method FreeTuner can support various personalized image generation: (a) subject-driven, (b) style-driven, and (c) compositional personalization.
• Figure 2: Given "A photo of a horse walking in Times Square", B-LoRA not only distorts the horse's structure but also fails to render the entire scene.
• Figure 3: Visualization of the image estimations corresponding to different timesteps within the denoising process (row 1) and our two-stage disentanglement strategy (row 2).
• Figure 4: Overview of the proposed FreeTuner. (a) In the preprocessing step, we get a binary mask $M_{sub}$ including only the subject through off-the-shelf models and inverse $I_{sub} * M_{sub}$ with a simple prompt $P_{sub}$ to acquire latent code $z_T^{sub}$. (b) Our generation process is divided into two stages. In the first stage, we focus on content generation which injects the intermediate features obtained from the reconstruction branch into the personalized branch. Upon entering the style generation stage, an additional visual encoder (e.g., VGG-19 vgg19) and guidance function will steer the generated image toward a similar style expressed in $I_{sty}$.
• Figure 5: Visualization of subject-related features: The top row displays the average CA maps for each word in the $P_{sub}$. In the bottom row, we perform PCA on latent codes $z$ across all diffusion steps and SA maps.