Forget Less by Learning Together through Concept Consolidation

TL;DR

This work tackles catastrophic forgetting in Custom Diffusion Models by proposing Forget Less by Learning Together (FL2T), a two-step, order-agnostic framework that uses set-invariant inter-concept guidance via proxy embeddings. In Step 1, each concept is learned independently with LoRA updates; in Step 2, transformer-based concept aggregation leverages proxies to capture higher-order inter-concept interactions and prompts, enabling positive knowledge transfer while reducing drift. The approach yields consistent improvements in CLIP-based Image Alignment and Text Alignment, identity preservation, and generation quality across CIFC, CelebA, and ImageNet, with efficiency gains in reference data usage and competitive parameter costs. Theoretical and empirical analyses support that unnormalized attention-based aggregation can reduce model drift, highlighting FL2T’s potential for scalable, multi-concept diffusion personalization.

Abstract

Custom Diffusion Models (CDMs) have gained significant attention due to their remarkable ability to personalize generative processes. However, existing CDMs suffer from catastrophic forgetting when continuously learning new concepts. Most prior works attempt to mitigate this issue under the sequential learning setting with a fixed order of concept inflow and neglect inter-concept interactions. In this paper, we propose a novel framework - Forget Less by Learning Together (FL2T) - that enables concurrent and order-agnostic concept learning while addressing catastrophic forgetting. Specifically, we introduce a set-invariant inter-concept learning module where proxies guide feature selection across concepts, facilitating improved knowledge retention and transfer. By leveraging inter-concept guidance, our approach preserves old concepts while efficiently incorporating new ones. Extensive experiments, across three datasets, demonstrates that our method significantly improves concept retention and mitigates catastrophic forgetting, highlighting the effectiveness of inter-concept catalytic behavior in incremental concept learning of ten tasks with at least 2% gain on average CLIP Image Alignment scores.

Figures (9)

• Figure 1: Forget Less by Learning Together (FL2T). Our method focuses on learning $G$ concepts in an order-agnostic incremental learning problem setting with fewer parameters and fewer reference images while also mitigating catastrophic forgetting. Unlike previous works, we leverage inter-concept interactions positively. The above image showcases examples of generated images, with a source image and an associated text prompt (left column) as input. We evaluate the generated image (right column) against the SOTA model CIDM cidm (middle column). The red and green arrows indicate regions of undesirable and desirable qualities, and their reasons are stated below each image.
• Figure 2: FL2T overview. We begin by independently training $G$ models on $G$ concepts for one epoch. Subsequently, in the second epoch, we utilize the learnt concept embeddings (from epoch one) across all concepts to perform cross-concept interactions or aggregate the concepts. To perform these interactions, we use the proxies (initialized with concept embeddings from epoch one) and two transformer layers. The goal of using transformer layers is to capture higher-order interactions between the concept embeddings, $g$-th concept embedding, and input prompt for task $g$. Subsequently, we compute a similarity matrix through matrix multiplication to weight the inter-concept interaction. This framework allows variable order of concept inflow, mitigates catastrophic forgetting, and outperforms SOTA frameworks with fewer parameters and fewer reference images.
• Figure 3: Ablation studies. We compare the performance of FL2T against CIDM: with a constraint on the (a) number of reference images, and (b) on the LoRA rank. In each case, FL2T exhibits superior performance.
• Figure 4: Scalability. FL2T shows impressive scalability achieving higher average CLIP and IMS scores (left axis) and lower FID values (right axis) over three concept ranges (1–10, 11–20, 21–30), compared to CIDM cidm on the ImageNet dataset imagenet. We limit the scalability analysis to 30 concepts due to CLIP tokenizer's limit on 77 new tokens.
• Figure 5: Qualitative Analysis on the CIFC dataset cidm. We compare the synthesized images by CIDM cidm and FL2T (Ours). The images are generated with a source image and an associated text prompt as input. Images with red and green arrows indicate regions of undesirable and desirable qualities, and their reasons are stated below each image.

Theorems & Definitions (4)

• Lemma A.1: Universal Upper Bound
• proof
• Theorem A.1: Existence of Reduced Drift
• proof