Personalize Anything for Free with Diffusion Transformer

TL;DR

This work shows that diffusion-transformer-based personalization can be achieved without any training by leveraging simple token replacement to anchor subject identity. By identifying DiT's position-disentangled token representations, the authors propose Personalize Anything, which uses timestep-adaptive replacement and patch perturbations to balance identity preservation with creative flexibility. The framework supports layout-guided generation, multi-subject composition, and mask-controlled editing, achieving state-of-the-art performance across diverse personalization tasks. Overall, the method offers a scalable, training-free path to high-fidelity personalized synthesis with broad practical impact for edited and composed imagery.

Abstract

Personalized image generation aims to produce images of user-specified concepts while enabling flexible editing. Recent training-free approaches, while exhibit higher computational efficiency than training-based methods, struggle with identity preservation, applicability, and compatibility with diffusion transformers (DiTs). In this paper, we uncover the untapped potential of DiT, where simply replacing denoising tokens with those of a reference subject achieves zero-shot subject reconstruction. This simple yet effective feature injection technique unlocks diverse scenarios, from personalization to image editing. Building upon this observation, we propose \textbf{Personalize Anything}, a training-free framework that achieves personalized image generation in DiT through: 1) timestep-adaptive token replacement that enforces subject consistency via early-stage injection and enhances flexibility through late-stage regularization, and 2) patch perturbation strategies to boost structural diversity. Our method seamlessly supports layout-guided generation, multi-subject personalization, and mask-controlled editing. Evaluations demonstrate state-of-the-art performance in identity preservation and versatility. Our work establishes new insights into DiTs while delivering a practical paradigm for efficient personalization.

Figures (14)

• Figure 1: Personalize Anything is a training-free framework based on Diffusion Transformers (DiT) for personalized image generation. The framework demonstrates advanced versatility, excelling in single-subject personalization (top), multi-subject or subject-scene composition, inpainting and outpainting (middle), as well as applications like visual storytelling (bottom), all without any training or fine-tuning.
• Figure 2: Simple token replacement in DiT (right) achieves high-fidelity subject reconstruction through its position-disentangled representation, while U-Net's convolutional entanglement (left) induces blurred edges and artifacts.
• Figure 3: Attention sharing consistoryFreeCustom fails in DiT due to the explicit positional encoding mechanism. When keeping the original positions $(i,j)\in [0,w)\times[0,h)$ in reference tokens, denoising tokens over-attend to reference ones with the same positions (shown in attention maps of (a)), resulting in ghosting artifacts in the generated image. Modified strategies, (b) removing positions and (c) shifting to non-overlapping regions, avoid collisions but loses identity alignment, as attention is almost absent on reference tokens.
• Figure 4: Method overview. Our framework anchors subject identity in early denoising through mask-guided token replacement with preserved positional encoding, and transitions to multi-modal attention for semantic fusion with text in later steps. During token replacement, we inject variations via patch perturbations. This timestep-adaptive strategy balances identity preservation and generative flexibility.
• Figure 5: Seamless extensions. Our framework enables: (a) layout-guided generation by translating token-injected regions, (b) multi-subject composition through sequential token injection, and (c) inpainting and outpainting via specifying masks and increased replacement.