Toward industrial use of continual learning : new metrics proposal for class incremental learning

TL;DR

The paper tackles the problem that Mean Task Accuracy (ACC) inadequately captures continual learning performance in industrial settings due to forgetting and task distribution, potentially leading to unsafe decisions. It proposes Minimal Incremental Class Accuracy (MICA) as a worst-case lower bound on per-class performance, defined by $MICA_i = \min_{c_k \in \cup_{j \le i} t_j} r_{ijk}$, to reveal weaknesses hidden by ACC. Building on MICA, it introduces Weighted Average Minimal Incremental Class Accuracy (WAMICA) with $w_T = 1 - (MICA_{max} - MICA_{min})$ and $WAMICA = \frac{w_T}{T} \sum_{j=1}^{T} MICA_j$ to capture both performance level and its variation across tasks. Validated on CIFAR10/100 with Gdumb, DER, and Weight Align, the results show that methods performing well under ACC may still exhibit poor worst-case or variable performance, underscoring the need for safer, industry-ready metrics that support risk-aware decision making ($MICA$, $WAMICA$). The work highlights practical implications for quality management and points to future directions that incorporate additional industrial metrics and methods to estimate forgetting without test data.

Abstract

In this paper, we investigate continual learning performance metrics used in class incremental learning strategies for continual learning (CL) using some high performing methods. We investigate especially mean task accuracy. First, we show that it lacks of expressiveness through some simple experiments to capture performance. We show that monitoring average tasks performance is over optimistic and can lead to misleading conclusions for future real life industrial uses. Then, we propose first a simple metric, Minimal Incremental Class Accuracy (MICA) which gives a fair and more useful evaluation of different continual learning methods. Moreover, in order to provide a simple way to easily compare different methods performance in continual learning, we derive another single scalar metric that take into account the learning performance variation as well as our newly introduced metric.

Figures (5)

• Figure 1: We investigate how the Mean Task Accuracy impacts industrial equipements performance. We find that this metrics underestimate the risk of failure and can lead to wrong choices of methods. We correct this problem with our new metric, Minimal Incremental Class Accuracy which gives a lower bound of the performance. Our metric guarantee fairness in CIL methods comparison and is safe for a quality management system in industrial cases.
• Figure 2: Mean Task Accuracy (ACC)
• Figure 3: Distribution of classes accuracies during training
• Figure 4: Minimum Incremental Class Accuracy ( MICA) (in transparent dashed, ACC)
• Figure 5: Weighted Average Minimal Incremental Class Accuracy (best viewed in color)