Premonition: Using Generative Models to Preempt Future Data Changes in Continual Learning

TL;DR

It is shown here that the combination of a large language model and an image generation model can similarly provide useful premonitions as to how a continual learning challenge might develop over time, thus becoming a valuable input to any existing continual learning method.

Abstract

Continual learning requires a model to adapt to ongoing changes in the data distribution, and often to the set of tasks to be performed. It is rare, however, that the data and task changes are completely unpredictable. Given a description of an overarching goal or data theme, which we call a realm, humans can often guess what concepts are associated with it. We show here that the combination of a large language model and an image generation model can similarly provide useful premonitions as to how a continual learning challenge might develop over time. We use the large language model to generate text descriptions of semantically related classes that might potentially appear in the data stream in future. These descriptions are then rendered using Stable Diffusion to generate new labelled image samples. The resulting synthetic dataset is employed for supervised pre-training, but is discarded prior to commencing continual learning, along with the pre-training classification head. We find that the backbone of our pre-trained networks can learn representations useful for the downstream continual learning problem, thus becoming a valuable input to any existing continual learning method. Although there are complexities arising from the domain gap between real and synthetic images, we show that pre-training models in this manner improves multiple Class Incremenal Learning (CIL) methods on fine-grained image classification benchmarks. Supporting code can be found at https://github.com/cl-premonition/premonition.

Figures (3)

• Figure 1: Overview of Premonition.Premonition involves first prompting GPT-4 openai2023gpt4 to produce a set of text prompts containing a class name and description within a semantic realm (in this example, we use birds as the realm). It then provides these prompts as inputs to Stable Diffusion, in order to generate a dataset of synthetic imagery. Premonition then adapts a pre-trained classification model by careful transfer learning on the synthetic dataset. The resulting backbone network becomes the input to an existing CL method, ideally one designed for pre-trained models zhou2024continual.
• Figure 2: Comparison of real images with synthetic images. The two images in the top row are for the species Pionus menstruus, obtained from iNaturalist-2018 inat. The two images in the bottom row are examples generated by Stable Diffusion from the prompt example given in \ref{['S:prompt']}. The second synthetic image is clearly a failure case, and many others were evident. Nevertheless, we found that Premonition produces benefits despite being trained on images that sometimes do not resemble the real concepts as well as they ideally should.
• Figure 3: Distribution of class counts in training splits of datasets. This figure illustrates that most of the datasets we demonstrate Premonition on have long tails where many classes have relatively few training samples.