IMAGE-ALCHEMY: Advancing subject fidelity in personalised text-to-image generation

TL;DR

The paper tackles the problem of personalizing diffusion-based text-to-image models to novel subjects using very few reference images, which often leads to catastrophic forgetting and high computational cost. It introduces Image-Alchemy, a two-stage pipeline that first learns a subject-specific, low-rank LoRA adaptation on SDXL's attention layers using rare placeholder tokens, and then inserts the subject into a generic SDXL-generated scene via a segmentation-guided Img2Img step. Key contributions include a token- and LoRA-based personalization framework, a segmentation-driven subject replacement, and empirical evidence showing a DINO similarity of 0.789 for the personalized subject with minimal degradation to the rest of the image. The approach preserves the base model’s broad generative capabilities while enabling high-fidelity subject integration, offering a practical and efficient path toward robust personalization in diffusion models.

Abstract

Recent advances in text-to-image diffusion models, particularly Stable Diffusion, have enabled the generation of highly detailed and semantically rich images. However, personalizing these models to represent novel subjects based on a few reference images remains challenging. This often leads to catastrophic forgetting, overfitting, or large computational overhead.We propose a two-stage pipeline that addresses these limitations by leveraging LoRA-based fine-tuning on the attention weights within the U-Net of the Stable Diffusion XL (SDXL) model. First, we use the unmodified SDXL to generate a generic scene by replacing the subject with its class label. Then, we selectively insert the personalized subject through a segmentation-driven image-to-image (Img2Img) pipeline that uses the trained LoRA weights.This framework isolates the subject encoding from the overall composition, thus preserving SDXL's broader generative capabilities while integrating the new subject in a high-fidelity manner. Our method achieves a DINO similarity score of 0.789 on SDXL, outperforming existing personalized text-to-image approaches.

Figures (6)

• Figure 1: Image-Alchemy : Illustration of personalized generation for the subject “iklan”, the token chosen for the given person. The left panel shows four reference images used to fine-tune our model, and the right panel demonstrates four diverse outputs, highlighting how the learned subject adapts seamlessly to various prompts.
• Figure 2: Overall Pipeline of Image-Alchemy
• Figure 3: Illustration of our Img2Img pipeline. The base image is first segmented and blurred (left), then encoded into the latent space. The learned LoRA token (e.g., “immen”) is combined with the prompt and fed to the U-Net for iterative refinement, after which the decoder produces the final, personalized image (right).
• Figure 4: Column 1 has the prompts used to generate the images, Column 2 portrays the images generated using normal fine-tuning technique, the Column 3 contains the corresponding final output images of our pipeline.
• Figure 5: Seamless Subject Integration. Row 1 displays a combination of bear and cat images, merging their characteristics while preserving the overall quality and context.Row 2 presents images of various women, each integrated smoothly while keeping their unique identities.Row 3 features images of different men, with each image tailored to an individual while maintaining coherence.Row 4 showcases images of dogs, each distinctly personalized to a different dog, while ensuring a consistent style and quality..