Low-Rank Continual Personalization of Diffusion Models

TL;DR

This work addresses catastrophic forgetting in continual personalization of diffusion models using LoRA adapters under a regime with no access to past adapters. It systematically compares four strategies—Naïve fine-tuning, Merge & Initialization, Merge & Orthogonal Initialization, and Magnitude-based merging—under the final model equation $W_{1\dots T} = W_0 + B_{1\dots T}A_{1\dots T}$ and shows that reinitializing LoRA weights before merging yields the best balance between plasticity and stability. The key contributions are a rigorous evaluation of initialization and merging techniques for continual diffusion-model customization and the finding that standard merge strategies can outperform naive approaches, with orthogonalization offering high plasticity and magnitude-based methods providing strong forgetting control. The results have practical implications for scalable, memory-efficient continual personalization of diffusion models in real-world applications, enabling sequential learning of new concepts without storing all past adapters.

Abstract

Recent personalization methods for diffusion models, such as Dreambooth and LoRA, allow fine-tuning pre-trained models to generate new concepts. However, applying these techniques across consecutive tasks in order to include, e.g., new objects or styles, leads to a forgetting of previous knowledge due to mutual interference between their adapters. In this work, we tackle the problem of continual customization under a rigorous regime with no access to past tasks' adapters. In such a scenario, we investigate how different adapters' initialization and merging methods can improve the quality of the final model. To that end, we evaluate the naive continual fine-tuning of customized models and compare this approach with three methods for consecutive adapters' training: sequentially merging new adapters, merging orthogonally initialized adapters, and updating only relevant task-specific weights. In our experiments, we show that the proposed techniques mitigate forgetting when compared to the naive approach. In our studies, we show different traits of selected techniques and their effect on the plasticity and stability of the continually adapted model. Repository with the code is available at https://github.com/luk-st/continual-lora.

Figures (8)

• Figure 1: Comparison of initialization and merging in four evaluated methods. The expressions in the adapters refer to how they are initialized with a new task. Numbers next to each schema refer to textual descriptions in \ref{['sec:methods']}.
• Figure 2: DINO score on the first task over continual fine-tuning on next tasks for the object and styles customization.
• Figure 3: Images generated by the model fine-tuned on 10 tasks. The first left column represents the subsequent tasks, each represented by an image illustrating the object to learn. We show how each method sequentially performs using the prompt templates: 'a $V^*$ wearing a santa hat' and 'a $V^*$ in a jail'.
• Figure 4: Images generated by the model fine-tuned on 10 tasks. The first left column represents the subsequent tasks, each represented by an image illustrating the style to learn. We show how each method sequentially performs using the prompt templates: '{style_name} image of a wizard' and '{style_name} image of a mirror'.
• Figure 5: Heatmap of CLIP-I alignment scores for each task in continual object personalization.