The Lie of the Average: How Class Incremental Learning Evaluation Deceives You?

TL;DR

This work introduces the concept of extreme sequences and provides theoretical justification for their crucial role in the reliable evaluation of CIL, and proposes EDGE (Extreme case-based Distribution&Generalization Evaluation), an evaluation protocol that adaptively identifies and samples extreme class sequences using inter-task similarity, offering a closer approximation of the ground-truth performance distribution.

Abstract

Class Incremental Learning (CIL) requires models to continuously learn new classes without forgetting previously learned ones, while maintaining stable performance across all possible class sequences. In real-world settings, the order in which classes arrive is diverse and unpredictable, and model performance can vary substantially across different sequences. Yet mainstream evaluation protocols calculate mean and variance from only a small set of randomly sampled sequences. Our theoretical analysis and empirical results demonstrate that this sampling strategy fails to capture the full performance range, resulting in biased mean estimates and a severe underestimation of the true variance in the performance distribution. We therefore contend that a robust CIL evaluation protocol should accurately characterize and estimate the entire performance distribution. To this end, we introduce the concept of extreme sequences and provide theoretical justification for their crucial role in the reliable evaluation of CIL. Moreover, we observe a consistent positive correlation between inter-task similarity and model performance, a relation that can be leveraged to guide the search for extreme sequences. Building on these insights, we propose EDGE (Extreme case-based Distribution & Generalization Evaluation), an evaluation protocol that adaptively identifies and samples extreme class sequences using inter-task similarity, offering a closer approximation of the ground-truth performance distribution. Extensive experiments demonstrate that EDGE effectively captures performance extremes and yields more accurate estimates of distributional boundaries, providing actionable insights for model selection and robustness checking. Our code is available at https://github.com/AIGNLAI/EDGE.

Figures (12)

• Figure 1: Existing CIL evaluations may be misleading! They merely compute the average accuracy without perceiving the performance distribution, failing to anticipate the impact of potential extreme sequences on the model.
• Figure 2: \ref{['fig: cifar', 'fig: imagenet']} show model performance under fully enumerable scenarios (green: maximum, red: minimum), along with estimates from the random sampling (RS) protocol (blue error bars). \ref{['fig: 23']} illustrates the correlation between inter-task similarity scores and model performance, where $R$ denotes the Pearson correlation coefficient.
• Figure 3: Illustration of the EDGE evaluation protocol. The sequence with a green background represents a hard case, where similar classes (e.g., apples and pears) appear within the same task, resulting in low inter-task similarity. The sequence with an orange background represents an easy case, where similar classes are distributed across different tasks, leading to high inter-task similarity.
• Figure 4: Effect of task sequences generated with CLIP text encoders of varying scales on the estimation of performance distributions under the EDGE protocol. The black curve denotes the ground-truth distribution, and the blue curve indicates the estimation obtained via the RS protocol.
• Figure 5: Visualization of the estimated lower and upper performance bounds across three datasets under the classic CIL setting: (a) CIFAR-100, (b) CUB-200, and (c) ImageNet-R. The slashed bars (/) denote the proposed EDGE, while the dotted bars (.) correspond to the existing RS protocol.

Theorems & Definitions (17)

• Lemma 1
• Theorem 1
• Remark 1
• Theorem 2
• Remark 2
• Theorem 3
• Lemma 1
• proof
• Theorem 1
• proof