Divide and not forget: Ensemble of selectively trained experts in Continual Learning

TL;DR

SEED tackles exemplar-free class-incremental learning by maintaining a fixed ensemble of experts that share early layers, but only one expert is finetuned per new task. Each expert represents classes with multivariate Gaussian distributions in its latent space, and a KL-based overlap criterion selects the updating expert to minimize representational drift. Inference uses a Bayesian ensemble over experts, and the selected expert is trained with cross-entropy plus distillation to preserve past knowledge. Across CIFAR-100, DomainNet, and ImageNet-Subset, SEED achieves state-of-the-art results, demonstrating strong plasticity and stability, as well as efficient parameter usage through shared layers and selective updating.

Abstract

Class-incremental learning is becoming more popular as it helps models widen their applicability while not forgetting what they already know. A trend in this area is to use a mixture-of-expert technique, where different models work together to solve the task. However, the experts are usually trained all at once using whole task data, which makes them all prone to forgetting and increasing computational burden. To address this limitation, we introduce a novel approach named SEED. SEED selects only one, the most optimal expert for a considered task, and uses data from this task to fine-tune only this expert. For this purpose, each expert represents each class with a Gaussian distribution, and the optimal expert is selected based on the similarity of those distributions. Consequently, SEED increases diversity and heterogeneity within the experts while maintaining the high stability of this ensemble method. The extensive experiments demonstrate that SEED achieves state-of-the-art performance in exemplar-free settings across various scenarios, showing the potential of expert diversification through data in continual learning.

Figures (11)

• Figure 1: Exemplar-free Class Incremental Learning methods evaluated on CIFAR100 divided into eleven tasks for two different data distributions.
• Figure 2: SEED comprises $K$ deep network experts $g_k \circ f$ (here $K=2$), sharing the initial layers $f$ for higher computational performance. $f$ are frozen after the first task. Each expert contains one Gaussian distribution per class $c \in C$ in his unique latent space. In this example, we consider four classes, classes $1$ and $2$ from task $1$ and classes $3$ and $4$ from task $2$. During inference, we generate latent representations of input $x$ for each expert and calculate its log-likelihoods for distributions of all classes (for each expert separately). Then, we softmax those log-likelihoods and compute their average over all experts. The class with the highest average softmax is considered as the prediction.
• Figure 3: SEED training process for $K=2$ experts, $T=3$ tasks, and $|C_t|=2$ classes per task. When the third task appears with novel classes $C_3$, we analyze distributions of $C_3$ classes (here represented as purple distributions) in latent spaces of all experts. We choose the expert where those distributions overlap least (here, expert 2). We finetune this expert to increase the separability of new classes further and move to the next task.
• Figure 4: Class incremental accuracy achieved after each task for equal splits on CIFAR100 and DomainNet. SEED significantly outperforms other methods in equal split scenarios for many tasks (left) and more considerable data shifts (right).
• Figure 5: Diversity of experts on CIFAR-100 dataset with $T=20$ split. The presented metric is relative accuracy (%) calculated by subtracting the accuracy of each expert from the averaged accuracy of all experts. Black squares represent experts selected to be finetuned on a given task. Although we do not impose any cost function associated with experts' diversity, they tend to specialize in different tasks by the design of our method. Moreover, our ensemble (bottom row) always performs better than the best expert, proving that each expert contributes uniquely to the ensemble in SEED.