MuseumMaker: Continual Style Customization without Catastrophic Forgetting

TL;DR

MuseumMaker tackles continual style customization for pre-trained text-to-image diffusion models under streaming, memory-constrained data. It introduces Style Distillation Loss ($\mathcal{L}_{\mathrm{SDL}}$) to focus learning on style while attenuating content overfitting, Dual Regularization for shared-LoRA ($\mathcal{L}_{\mathrm{DR}}$) to stabilize updates across weight and feature spaces, and Task-wise Token Learning (TTL) to store per-style tokens across cross-attention layers. Across WikiArt styles, MuseumMaker shows superior FID and CLIP scores and reduced forgetting compared to finetuning, EWC, and LWF baselines, while maintaining a compact parameter footprint (39.06M) and reasonable training time. The results also demonstrate effective style transfer and practical applicability for building a private style museum, with robust performance under a never-ending style stream. The proposed framework advances practical continual style learning for diffusion models by balancing plasticity and stability with efficient, modular components.

Abstract

Pre-trained large text-to-image (T2I) models with an appropriate text prompt has attracted growing interests in customized images generation field. However, catastrophic forgetting issue make it hard to continually synthesize new user-provided styles while retaining the satisfying results amongst learned styles. In this paper, we propose MuseumMaker, a method that enables the synthesis of images by following a set of customized styles in a never-end manner, and gradually accumulate these creative artistic works as a Museum. When facing with a new customization style, we develop a style distillation loss module to extract and learn the styles of the training data for new image generation. It can minimize the learning biases caused by content of new training images, and address the catastrophic overfitting issue induced by few-shot images. To deal with catastrophic forgetting amongst past learned styles, we devise a dual regularization for shared-LoRA module to optimize the direction of model update, which could regularize the diffusion model from both weight and feature aspects, respectively. Meanwhile, to further preserve historical knowledge from past styles and address the limited representability of LoRA, we consider a task-wise token learning module where a unique token embedding is learned to denote a new style. As any new user-provided style come, our MuseumMaker can capture the nuances of the new styles while maintaining the details of learned styles. Experimental results on diverse style datasets validate the effectiveness of our proposed MuseumMaker method, showcasing its robustness and versatility across various scenarios.

Figures (8)

• Figure 1: Motivation of our proposed MuseumMaker model. The desired style (such as impressionist or cubism) for each generated text-to-image task can be customized by the user with a few images. Our MuseumMaker model can continually incorporate the diverse styles without catastrophic forgetting, and further accumulate these creative works as a private Museum.
• Figure 2: Illustration of our proposed continual style customization for T2I diffusion model, i.e., MuseumMaker. It mainly contains a Dual Regularization for Shared-LoRA (DR-LoRA) module to regularize the optimization of model from both weight and feature aspects, a Task-wise Token Learning (TTL) module to store the text embedding of each style learning to reduce forgetting and a Style Distillation Loss module (SDL) to make the model focus on the style of learning images.
• Figure 3: Qualitative comparison between our method with competing methods in 10 tasks continual style adaption setting, where the first two rows represent the stylistic dataset and prompts provides users, and the rest results denote the image generated by each method with the same text prompt, i.e., a cat, wearing a sunglasses in *style.
• Figure 4: Qualitative comparison of our ablation studies, where we evaluate the contribution of each module we proposed. We denotes the task-wise token learning module as TTL and style distillation loss as SDL in the fig, respectively. We input the same text prompt as we do in comparison experiments. The upper bound setting stores a learnable token embedding and LoRA weight for each style
• Figure 5: Style loss, FID and CLIP score for continual style adaptation setting. The red arrow in the image indicates the direction where the score improves. Comparing with other competing method (e.g., LoRAb8+LWF, LoRA+EWC, SPD+LWF, SPD+EWC), our MuseumMaker achieves the best performance in terms of FID and CLIP score.