Just Say the Name: Online Continual Learning with Category Names Only via Data Generation

TL;DR

A diverse prompt generation method, HIerarchical Recurrent Prompt Generation (HIRPG) as well as COmplexity-NAvigating eNsembler (CONAN) that selects samples with minimal overlap from multiple generative models for the name only continual learning.

Abstract

Requiring extensive human supervision is often impractical for continual learning due to its cost, leading to the emergence of 'name-only continual learning' that only provides the name of new concepts (e.g., classes) without providing supervised samples. To address the task, recent approach uses web-scraped data but results in issues such as data imbalance, copyright, and privacy concerns. To overcome the limitations of both human supervision and webly supervision, we propose Generative name only Continual Learning (GenCL) using generative models for the name only continual learning. But naïve application of generative models results in limited diversity of generated data. So, we specifically propose a diverse prompt generation method, HIerarchical Recurrent Prompt Generation (HIRPG) as well as COmplexity-NAvigating eNsembler (CONAN) that selects samples with minimal overlap from multiple generative models. We empirically validate that the proposed GenCL outperforms prior arts, even a model trained with fully supervised data, in various tasks including image recognition and multi-modal visual reasoning. Data generated by GenCL is available at https://anonymous.4open.science/r/name-only-continual-E079.

Figures (14)

• Figure 1: Comparison of Manually Annotated (MA) data, Web-Scraped Data, and Generated data. Generated data addresses constraints associated with Web-scraped or MA data, mitigating privacy concerns and usage restrictions (i.e., whether images can be used for learning). Also, it maintains controllability (the ability to generate images with various contexts, e.g., background, color) as desired. Generated data are less noisy (i.e., containing fewer undesired images) than web-scrapped data and proves to be a more cost-effective than MA data which requires human annotation. For more details on the terminology employed in this figure, see Sec. \ref{['app:terms']}
• Figure 2: Illustration of the proposed GenCL framework. When a new concept that needs to be learned is encountered, it is passed through a prompt generation module, $\psi$, to produce diverse prompts. These prompts are then used to generate data from a set of generators, $\mathcal{G}$. The data generated by each generator are combined through the ensembler, $\Delta$, and subsequently used to train the model, $f_\theta$.
• Figure 3: Samples with high RMD scores and low RMD scores
• Figure 4: An example of the Bongard-HOI task. CA refers to the concept answering task, while P/N refers to the classifying whether a query image belongs to the positive or negative set.
• Figure 5: An example of the Bongard-OpenWorld task. CA refers to the concept answering task, while P/N refers to the classifying whether a query image belongs to the positive or negative set. The concept $c$ is free-form, such as sentences.