CLoG: Benchmarking Continual Learning of Image Generation Models

TL;DR

This work formalizes Continual Learning of Generative models (CLoG) and argues for shifting the CL focus from classification to generative tasks. It establishes a foundational benchmark framework, including seven label-conditioned and one concept-conditioned CLoG tasks, and adapts twelve baselines across regularization, replay, and parameter-isolation families to two generative backbones (GAN and diffusion models). The study provides unified evaluation metrics (AIQ, AFQ, FR) and standardized training protocols, uncovering core challenges such as mode collapse in replay methods and limited transfer in parameter-isolation approaches, with diffusion models generally offering stronger performance. By releasing a public codebase and offering insights into task design, evaluation, and efficiency, the paper lays groundwork for future CLoG methods and highlights the importance of lifelong learning in next-generation AI-generated content.

Abstract

Continual Learning (CL) poses a significant challenge in Artificial Intelligence, aiming to mirror the human ability to incrementally acquire knowledge and skills. While extensive research has focused on CL within the context of classification tasks, the advent of increasingly powerful generative models necessitates the exploration of Continual Learning of Generative models (CLoG). This paper advocates for shifting the research focus from classification-based CL to CLoG. We systematically identify the unique challenges presented by CLoG compared to traditional classification-based CL. We adapt three types of existing CL methodologies, replay-based, regularization-based, and parameter-isolation-based methods to generative tasks and introduce comprehensive benchmarks for CLoG that feature great diversity and broad task coverage. Our benchmarks and results yield intriguing insights that can be valuable for developing future CLoG methods. Additionally, we will release a codebase designed to facilitate easy benchmarking and experimentation in CLoG publicly at https://github.com/linhaowei1/CLoG. We believe that shifting the research focus to CLoG will benefit the continual learning community and illuminate the path for next-generation AI-generated content (AIGC) in a lifelong learning paradigm.

Figures (14)

• Figure 1: Overview of benchmarks. Seven label-conditioned and one concept-conditioned CLoG benchmarks are studied, with details presented in \ref{['tab.dataset_comparison']} and \ref{['sec:task_selection']}. Label-conditioned CLoG learns a sequence of generation tasks conditioned on label indices. Concept-conditional CLoG learns to synthesize a sequence of concepts (denoted as $V_i^*$ for the $i$th concept) given arbitrary text prompts.
• Figure 2: Overview of baselines. Three types of CL baselines are adapted to CLoG, which include regularization-based, replay-based, and parameter-isolation-based methods, resulting in a total of twelve different CLoG baselines. The detailed information on the baselines are in \ref{['sec:baseline_setup']}.
• Figure B.1: Visualization results of label-conditioned CLoG on the MNIST MNIST dataset.
• Figure B.2: Visualization results of label-conditioned CLoG on the Fashion-MNIST fashionmnist dataset.
• Figure B.3: Visualization results of label-conditioned CLoG on the CIFAR-10 cifar10 dataset.