From Isolation to Integration: Building an Adaptive Expert Forest for Pre-Trained Model-based Class-Incremental Learning

TL;DR

The Semantic-guided Adaptive Expert Forest (SAEF) is proposed, a new method that organizes adapters into a structured hierarchy for better knowledge sharing and achieves SOTA performance.

Abstract

Class-Incremental Learning (CIL) requires models to learn new classes without forgetting old ones. A common method is to freeze a pre-trained model and train a new, lightweight adapter for each task. While this prevents forgetting, it treats the learned knowledge as a simple, unstructured collection and fails to use the relationships between tasks. To this end, we propose the Semantic-guided Adaptive Expert Forest (SAEF), a new method that organizes adapters into a structured hierarchy for better knowledge sharing. SAEF first groups tasks into conceptual clusters based on their semantic relationships. Then, within each cluster, it builds a balanced expert tree by creating new adapters from merging the adapters of similar tasks. At inference time, SAEF finds and activates a set of relevant experts from the forest for any given input. The final prediction is made by combining the outputs of these activated experts, weighted by how confident each expert is. Experiments on several benchmark datasets show that SAEF achieves SOTA performance.

Figures (7)

• Figure 1: Conceptual comparison of adapter-based CIL methods. (a) The conventional methods treat adapters as an unstructured pool, leading to inefficient full-ensemble inference. (b) Our proposed SAEF automatically organizes adapters into a semantic expert forest, enabling an efficient, adaptive search for the most relevant experts.
• Figure 2: Overview of the SAEF. Our method organizes individually trained task adapters into a structured knowledge hierarchy. (1) Stage 1: Conceptual Clustering. Based on semantic prototypes, tasks are grouped into high-level conceptual clusters (e.g., 'Animal', 'Vehicle'). (2) Stage 2: Hierarchical Construction. Within each cluster, a balanced expert tree is built via a bottom-up process of recursively fusing the most visually similar nodes. The roots of these trees are then merged to form a single Global Expert. (3) Stage 3: Adaptive Inference. For a given input, SAEF performs a parallel, multi-path search to dynamically activate a set of relevant experts. The final prediction is derived from an entropy-guided fusion of their outputs.
• Figure 3: Comparison against rehearsal-based methods. Average accuracy ($\bar{\mathcal{A}}$) is plotted as a function of the number of learned tasks. Rehearsal-based methods use 20 exemplars per class. Despite being an exemplar-free method, SAEF consistently outperforms strong rehearsal-based competitors across multiple benchmarks, and the performance gap often widens over time, demonstrating superior knowledge retention and scalability.
• Figure 4: Ablation and structural analysis of SAEF on ImageNet-R. Performance is measured by average accuracy ($\bar{\mathcal{A}}$). (a) Ablation study demonstrating the progressive contribution of each core component. (b) Structural analysis validating our balanced forest structure ($K=2$) over less structured alternatives.
• Figure 5: efficiency and performance evaluation on CIFAR-100. We plot average accuracy ($\bar{\mathcal{A}}$) against Training Throughput (samples/s). The size of each bubble is proportional to the Inference Time (ms/sample). The ideal method occupies the top-right corner with the smallest bubble, representing high accuracy, fast training, and rapid inference.