Continual HyperTransformer: A Meta-Learner for Continual Few-Shot Learning

TL;DR

The paper introduces Continual HyperTransformer (CHT), a meta-learned, Transformer-based hypernetwork that generates task-specific CNN weights from a live support set and recursively incorporates previously generated weights to learn a sequence of few-shot tasks without forgetting. To address interference across tasks, it replaces cross-entropy with a prototypical loss that stores class representations as task-aware prototypes $c_{\tau k}$, enabling fixed embedding dimensionality and robust cross-task generalization. Empirical results on Omniglot and tieredImageNet demonstrate strong performance across mini-batch, task-incremental, and class-incremental scenarios, with evidence of positive backward transfer and good extrapolation to more tasks. The approach supports rapid, on-device adaptation after meta-training and shows promising applicability to multi-domain sequences while maintaining privacy advantages through weight-based information sharing.

Abstract

We focus on the problem of learning without forgetting from multiple tasks arriving sequentially, where each task is defined using a few-shot episode of novel or already seen classes. We approach this problem using the recently published HyperTransformer (HT), a Transformer-based hypernetwork that generates specialized task-specific CNN weights directly from the support set. In order to learn from a continual sequence of tasks, we propose to recursively re-use the generated weights as input to the HT for the next task. This way, the generated CNN weights themselves act as a representation of previously learned tasks, and the HT is trained to update these weights so that the new task can be learned without forgetting past tasks. This approach is different from most continual learning algorithms that typically rely on using replay buffers, weight regularization or task-dependent architectural changes. We demonstrate that our proposed Continual HyperTransformer method equipped with a prototypical loss is capable of learning and retaining knowledge about past tasks for a variety of scenarios, including learning from mini-batches, and task-incremental and class-incremental learning scenarios.

Figures (15)

• Figure 1: In continual few-shot learning, the model learns from $T$ tasks sequentially. For the first task (task $0$), the CNN weights $\theta_0$ are generated using only the support set ${S^{(0)}}$. For each subsequent task $t$, the Continual HyperTransformer (CHT) uses the support set ${S^{(t)}}$ and the previously generated weights $\theta_{t-1}$ to generate the weights $\theta_{t}$. To update the weights $\psi$ of the CHT, the loss is calculated by summing the individual losses computed for each generated weight $\theta_{t}$ when evaluated on the query set of all the prior tasks $(Q^{(\tau)})_{\tau=0}^T$.
• Figure 2: The information flow of the HyperTransformer (HT) model (left) compared to the proposed Continual HyperTransformer (CHT) model (right). In the original HT, the input weight embeddings are initialized with empty placeholders. In contrast, the proposed CHT model incorporates information from past tasks when generating weights for the current task. The weight slice information from previously learned tasks is passed as input to the new iteration of the CHT. The CHT uses the support set for the current task and the input weight information to generate the weights. This allows the CHT to retain knowledge about past tasks and avoid forgetting when learning new tasks.
• Figure 3: Task-incremental learning on Omniglot and tieredImageNet. Each column represents a different CHT trained with a total of $T=2$, $3$, $4$ or $5$ tasks. The tasks marked with a bullet symbol ($\bullet$) correspond to the terms in the objective function \ref{['eq:ht-cl-all-tasks']} that are being minimized. The lines marked with the diamond symbol ($\diamond$) show the extrapolation of the trained CHT to a larger number of tasks. The confidence intervals do not exceed 0.5%.
• Figure 4: Class-incremental learning on Omniglot and tieredImageNet. Each column represents a different CHT trained with a total of $T=2$, $3$, $4$ or $5$ tasks. The tasks marked with a bullet symbol ($\bullet$) correspond to the terms in the objective function \ref{['eq:ht-cl-all-tasks']} that are being minimized. The lines marked with the diamond symbol ($\diamond$) show the extrapolation of the trained CHT to a larger number of tasks. The confidence intervals do not exceed 0.5%.
• Figure 5: The accuracy of the HT model trained for $T=5$ using the cross-entropy loss. The accuracy of the first weight $\theta_0$ is high and is better than the accuracy of the ConstPN model's embeddings. However, when more tasks are added, the accuracy drops dramatically due to collisions between the same classes for different tasks in the cross-entropy loss.