T-LoRA: Single Image Diffusion Model Customization Without Overfitting

TL;DR

Single-image diffusion personalization suffers from overfitting and limited diversity. T-LoRA combines a timestep-dependent rank masking strategy with orthogonal initialization (Ortho-LoRA) to allocate modeling capacity across diffusion timesteps while preserving information flow. Across SD-XL and FLUX-1.dev, T-LoRA demonstrates superior text alignment and robust concept fidelity relative to LoRA and other baselines, including in multi-image and user studies. This work highlights the practical value of timestep-aware adaptation and orthogonality in data-efficient diffusion personalization, with potential to inform broader adaptation strategies for generative models.

Abstract

While diffusion model fine-tuning offers a powerful approach for customizing pre-trained models to generate specific objects, it frequently suffers from overfitting when training samples are limited, compromising both generalization capability and output diversity. This paper tackles the challenging yet most impactful task of adapting a diffusion model using just a single concept image, as single-image customization holds the greatest practical potential. We introduce T-LoRA, a Timestep-Dependent Low-Rank Adaptation framework specifically designed for diffusion model personalization. We show that higher diffusion timesteps are more prone to overfitting than lower ones, necessitating a timestep-sensitive fine-tuning strategy. T-LoRA incorporates two key innovations: (1) a dynamic fine-tuning strategy that adjusts rank-constrained updates based on diffusion timesteps, and (2) a weight parametrization technique that ensures independence between adapter components through orthogonal initialization. Extensive experiments on SD-XL and FLUX-1.dev show that T-LoRA and its individual components outperform standard LoRA and other diffusion model personalization techniques, achieving a superior balance between concept fidelity and text alignment. Project page is available at https://controlgenai.github.io/T-LoRA/.

Figures (12)

• Figure 1: T-LoRA reduces overfitting related to position and background, enabling versatile and enriched generation.
• Figure 2: Motivational Experiment. The figure shows fine-tuning the SD-XL model with LoRA over fixed timestep intervals. Focusing on the noisiest timesteps causes rapid overfitting, affecting positioning and backgrounds. Shifting to earlier timesteps improves text alignment and generation flexibility. However, completely excluding the noisiest timesteps ($t \in [800; 1000]$) is infeasible, as they are essential for maintaining concept fidelity.
• Figure 3: Comparison of training methods: LoRA, the proposed Vanilla T-LoRA, and T-LoRA schemes.
• Figure 4: Singular values of B matrices for LoRA and Ortho-LoRA with $r=64$ after 800 training steps. The red line marks the rank capturing 95% of the total singular value sum. LoRA matrices are effectively low-rank -- especially in cross-attention -- while Ortho-LoRA maintains full rank.
• Figure 5: Results for six Ortho-LoRA initialization variants based on principal, middle, and last SVD components of the original weights $W$, and a random matrix $R$. Higher singular values correlate with overfitting, while too small values can slow down the training. Initialization with last singular values from $R$ is optimal across most ranks.