Forget Less by Learning from Parents Through Hierarchical Relationships

TL;DR

This work tackles catastrophic forgetting in Custom Diffusion Models by introducing FLLP, a continual-learning framework that leverages hyperbolic, parent-child concept hierarchies to positively transfer knowledge across sequential concept requests. By embedding concept representations on a Lorentzian manifold and enforcing parent entailment through a Hyperbolic Parent Entailment Loss, FLLP guides new concepts to arise within the established hierarchical structure, while storing representative attention maps to maintain prior reasoning. The method is implemented with a transfer-to-hyperboloid step, a parent-search procedure, and a hierarchical training objective that balances reconstruction loss with entailment constraints, yielding state-of-the-art results on CIFC, CelebA, and ImageNet benchmarks and demonstrating reduced parameter drift. Empirically, FLLP achieves robust preservation of earlier concepts and improved generalization when incorporating multiple new concepts, highlighting the practical value of structured inter-concept interactions in generative continual learning.

Abstract

Custom Diffusion Models (CDMs) offer impressive capabilities for personalization in generative modeling, yet they remain vulnerable to catastrophic forgetting when learning new concepts sequentially. Existing approaches primarily focus on minimizing interference between concepts, often neglecting the potential for positive inter-concept interactions. In this work, we present Forget Less by Learning from Parents (FLLP), a novel framework that introduces a parent-child inter-concept learning mechanism in hyperbolic space to mitigate forgetting. By embedding concept representations within a Lorentzian manifold, naturally suited to modeling tree-like hierarchies, we define parent-child relationships in which previously learned concepts serve as guidance for adapting to new ones. Our method not only preserves prior knowledge but also supports continual integration of new concepts. We validate FLLP on three public datasets and one synthetic benchmark, showing consistent improvements in both robustness and generalization.

Figures (14)

• Figure 1: Continual Learning for Concept Personalization and Hyperbolic Geometry. (a) The Continual Learning (CL) paradigm involves training a model on a sequence of tasks or concepts (e.g., Bulbul, Zebra, Chow-chow) over time. A major challenge in this setup is Catastrophic Forgetting, where the model tends to lose knowledge of previously learned concepts as it acquires new ones. (b) In Euclidean space, all concepts coexist without any inherent hierarchy. In contrast, hyperbolic geometry enables hierarchical representation of concepts. Importantly, hyperbolic distances (blue line) are fundamentally different from Euclidean distances (green line).
• Figure 2: Quantitative Analysis of 1D Gaussian. The length of the orange arrows reflects the forgetting rate for each concept, with shorter arrows indicating better retention. CIDM cidm improves upon the baseline by achieving a lower forgetting rate of 13.2. FLLP (Ours) further reduces this to 11.4. The visibly shorter orange arrows in the bottom row corroborate the improved knowledge retention of FLLP over CIDM. (Best viewed by zooming in)
• Figure 3: Qualitative Analysis. We compare the generated images in TI dosovitskiy2021imageworth16x16words, CIDM cidm and FLLP (Ours) on CelebA celeba and CIFC cidm. The red and green arrows indicate regions of undesirable and desirable qualities, and their reasons are stated below each image. The hyperbolic tree indicates the parent chain of concepts that the model traversed in learning the incoming concept.
• Figure 4: Ablation Studies. (a) FLLP scales to 35 concepts (limited by CLIP tokenizer) and consistently outperforms CIDM cidm on both IA and TA metrics. (b) Hyperbolic Parent Entailment Loss on LoRA weights benefits IA scores, but hurts TA scores. Thereby making Image Attention Maps (IAM) better suited for the loss function. (c) FLLP exhibits 22% lower parameter drift than CIDM, indicating the utilization of inter-concept interactions positively.
• Figure 5: Average CLIP scores of all ten concepts. Effect of threshold parameter on the average CLIP Score across the three datasets.