Continual Learning of Personalized Generative Face Models with Experience Replay

TL;DR

This work tackles open-world continual learning for personalized 2D and 3D generative face models by addressing catastrophic forgetting as new appearance batches arrive. It introduces two replay-based strategies—ER-Rand and ER-Hull—with ER-Hull leveraging a convex hull in StyleGAN's latent space to select the most informative past samples when buffer storage is limited. Across five identities and ten timestamps, ER-Hull consistently reduces forgetting relative to random replay and approaches Upper Bound performance at larger buffers, with pronounced gains in small-buffer regimes. The findings advance practical, open-world deployment of personalized facial priors by enabling continual adaptation while preserving past identity representations.

Abstract

We introduce a novel continual learning problem: how to sequentially update the weights of a personalized 2D and 3D generative face model as new batches of photos in different appearances, styles, poses, and lighting are captured regularly. We observe that naive sequential fine-tuning of the model leads to catastrophic forgetting of past representations of the individual's face. We then demonstrate that a simple random sampling-based experience replay method is effective at mitigating catastrophic forgetting when a relatively large number of images can be stored and replayed. However, for long-term deployment of these models with relatively smaller storage, this simple random sampling-based replay technique also forgets past representations. Thus, we introduce a novel experience replay algorithm that combines random sampling with StyleGAN's latent space to represent the buffer as an optimal convex hull. We observe that our proposed convex hull-based experience replay is more effective in preventing forgetting than a random sampling baseline and the lower bound.

Figures (4)

• Figure 1: Open-world deployment and training of personalized generative models nitzan2022mystyleqi2023my3dgen are challenging as images captured at each time has limited diversity in style, appearance, and lighting conditions. Naive finetuning of the model on each timestamp (Lower bound) leads to catastrophic forgetting, where the final model (t=10) performs poorly on test samples from previous timestamps. The Upper bound model is finetuned using all training images from all timestamps. We present an Experience Replay-based Continual Learning technique where we update a replay buffer to store the most informative images from the past as new images are captured at every timestamp. Our proposed technique, ER-Hull optimizes the buffer (size=3) as the most informative convex hull in StyleGAN's latent space.
• Figure 2: Diagram of the ER-Rand algorithm. We randomly sample the next replay buffer from the combination of the just-seen timestamp's data plus the replay buffer, and we only consider samples where we have the maximum possible number of timestamps represented.
• Figure 3: Illustration of ER-Hull. We perform RANSAC over many different possible replay buffers and choose the one that creates a convex hull that is "closest" to the other points, normalizing by timestamp.
• Figure 4: Performance deterioration of the final personalized (a) 2D generative model and (b) 3D generative model trained at t=10 on all previous time, averaged over 5 celebrities for 2D and 4 celebrities for 3D. ER-Hull outperforms ER-Rand on earlier timestamps proving its effectiveness in reducing forgetting.