Language-Inspired Relation Transfer for Few-shot Class-Incremental Learning

TL;DR

This work addresses Few-Shot Class-Incremental Learning by injecting pretrained language knowledge into the visual domain. It introduces Language-Inspired Relation Transfer (LRT), which uses a language-guided graph to transform text relationships into visual prototypes and a text-vision prototypical fusion to combine modalities. To cope with domain gaps and limited text data, the method adds context prompt learning for rapid adaptation and imagined contrastive learning to enrich multimodal alignment. Empirical results on miniImageNet, CIFAR-100, and ImageNet100 show substantial improvements over state-of-the-art baselines, highlighting the practical value of language-guided continual learning for vision tasks.

Abstract

Depicting novel classes with language descriptions by observing few-shot samples is inherent in human-learning systems. This lifelong learning capability helps to distinguish new knowledge from old ones through the increase of open-world learning, namely Few-Shot Class-Incremental Learning (FSCIL). Existing works to solve this problem mainly rely on the careful tuning of visual encoders, which shows an evident trade-off between the base knowledge and incremental ones. Motivated by human learning systems, we propose a new Language-inspired Relation Transfer (LRT) paradigm to understand objects by joint visual clues and text depictions, composed of two major steps. We first transfer the pretrained text knowledge to the visual domains by proposing a graph relation transformation module and then fuse the visual and language embedding by a text-vision prototypical fusion module. Second, to mitigate the domain gap caused by visual finetuning, we propose context prompt learning for fast domain alignment and imagined contrastive learning to alleviate the insufficient text data during alignment. With collaborative learning of domain alignments and text-image transfer, our proposed LRT outperforms the state-of-the-art models by over $13\%$ and $7\%$ on the final session of mini-ImageNet and CIFAR-100 FSCIL benchmarks.

Figures (11)

• Figure 1: The motivation of the proposed approach. Visual encoders provide clear boundaries in a) when learning with base sufficient data, while resulting in confused prototypes with only a few samples of novel classes in b). Our proposed LRT aims to transfer the pretrained language relationships to help construct a joint feature representation of both base and novel classes.
• Figure 2: Illustrations of different learning paradigms. a) Prototypical FSCIL zhang2021fewzhou2022forward: using visual prototypes for incremental classes. b) Zero-shot CLIP radford2021learning: direct predicting probabilities after image-text contrastive learning. c) Ours: transferring the pretrained text embedding to visual domains meanwhile keeping domain alignment with context prompt and imagined contrastive loss.
• Figure 3: The proposed Language-inspired Relation Transfer (LRT) approach consists of two essential modules. 1) Relational knowledge transfer module first transfers the text-wise relationship to the visual prototypes and a text-vision prototypical fusion module for knowledge fusion. 2) Image-Text alignment module introduces context prompt learning for fast adaptation and proposes the imagined contrastive learning for multi-modal alignment in few-shot class incremental learning.
• Figure 4: The motivation and modules of the proposed LRT. Our proposed LRT is composed of an aligning stage to conduct a multimodal alignment with few-shot downstream data and a transferring stage to transfer the text knowledge to the vision domain.
• Figure 5: Illustrations of different text-vision learning loss. a) Class-wise context prompt Learning. b) Multi-modality imagined contrastive learning: two images using mixing strategy yun2019cutmix are aligned with their corresponding prompt fusion texts.