FeNeC: Enhancing Continual Learning via Feature Clustering with Neighbor- or Logit-Based Classification

TL;DR

The paper tackles exemplar-free class-incremental learning with frozen feature extractors by modeling intra-class variability through multi-centroid per-class representations. FeNeC uses a Mahalanobis-distance based weighted kNN over class centroids, while FeNeC-Log adds a light-weight log-likelihood classifier with two shared parameters trained on the initial task. Both variants generalize FeCAM and achieve state-of-the-art or competitive performance across multiple benchmarks and backbones, with notable gains on CIFAR-100 and ImageNet-Subset and robust behavior under task sequences. Mahalanobis-distance based decisions consistently outperform Euclidean distance, and the proposed methods incur modest runtime overhead, offering an effective, memory-efficient approach to continual learning. This work advances memory-efficient continual learning by better capturing intra-class structure without replay.

Abstract

The ability of deep learning models to learn continuously is essential for adapting to new data categories and evolving data distributions. In recent years, approaches leveraging frozen feature extractors after an initial learning phase have been extensively studied. Many of these methods estimate per-class covariance matrices and prototypes based on backbone-derived feature representations. Within this paradigm, we introduce FeNeC (Feature Neighborhood Classifier) and FeNeC-Log, its variant based on the log-likelihood function. Our approach generalizes the existing concept by incorporating data clustering to capture greater intra-class variability. Utilizing the Mahalanobis distance, our models classify samples either through a nearest neighbor approach or trainable logit values assigned to consecutive classes. Our proposition may be reduced to the existing approaches in a special case while extending them with the ability of more flexible adaptation to data. We demonstrate that two FeNeC variants achieve competitive performance in scenarios where task identities are unknown and establish state-of-the-art results on several benchmarks.

Figures (3)

• Figure 1: The workflow of the methods presented in this paper. Initially, $k$--means clustering is performed on features extracted from the neural network. Then, in FeNeC, a kNN classifier with Mahalanobis distance is used, while in FeNeC-Log, each class has a corresponding logit value with shared parameters $a, b \in \mathbb{R}$, trained only during the initial task.
• Figure 2: The comparison of FeCAM, FeNeC and FeNeC-Log across incremental tasks on different datasets. The results were averaged over three different class orders (and random initializations for ResNet) for all methods.
• Figure 3: The last task accuracy for different hyperparameter values of FeNeC with ResNet-18 architecture and the ImageNet-Subset dataset. The best hyperparameters are marked with red lines. In the graphs comparing accuracy with hyperparameters, the color indicates the number of clusters, while in the graph comparing $N_{points}$ or $N_{neighbors}$ with $N_{clusters}$, the color indicates accuracy.