Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Don't forget, there is more than forgetting: new metrics for Continual Learning

Natalia Díaz-Rodríguez, Vincenzo Lomonaco, David Filliat, Davide Maltoni

TL;DR

The paper tackles the evaluation gap in continual learning by introducing an implementation-independent metric framework that extends beyond forgetting. It defines seven criteria—Accuracy over time, Backward and Forward Transfer (with REM and BWT+), Model Size, Samples Storage, and Computational Efficiency—normalized via MAVT and aggregated into a flexible CL_score, plus a stability measure. An empirical study on iCIFAR-100 demonstrates how CL_score can reveal tradeoffs between accuracy, memory, and compute under different weighting schemes, using common CL baselines (Naïve, Cumulative, EWC, LWF, SI). The work advocates for comprehensive, application-aware evaluation in CL and suggests directions for refining metrics in future research.

Abstract

Continual learning consists of algorithms that learn from a stream of data/tasks continuously and adaptively thought time, enabling the incremental development of ever more complex knowledge and skills. The lack of consensus in evaluating continual learning algorithms and the almost exclusive focus on forgetting motivate us to propose a more comprehensive set of implementation independent metrics accounting for several factors we believe have practical implications worth considering in the deployment of real AI systems that learn continually: accuracy or performance over time, backward and forward knowledge transfer, memory overhead as well as computational efficiency. Drawing inspiration from the standard Multi-Attribute Value Theory (MAVT) we further propose to fuse these metrics into a single score for ranking purposes and we evaluate our proposal with five continual learning strategies on the iCIFAR-100 continual learning benchmark.

Don't forget, there is more than forgetting: new metrics for Continual Learning

TL;DR

The paper tackles the evaluation gap in continual learning by introducing an implementation-independent metric framework that extends beyond forgetting. It defines seven criteria—Accuracy over time, Backward and Forward Transfer (with REM and BWT+), Model Size, Samples Storage, and Computational Efficiency—normalized via MAVT and aggregated into a flexible CL_score, plus a stability measure. An empirical study on iCIFAR-100 demonstrates how CL_score can reveal tradeoffs between accuracy, memory, and compute under different weighting schemes, using common CL baselines (Naïve, Cumulative, EWC, LWF, SI). The work advocates for comprehensive, application-aware evaluation in CL and suggests directions for refining metrics in future research.

Abstract

Continual learning consists of algorithms that learn from a stream of data/tasks continuously and adaptively thought time, enabling the incremental development of ever more complex knowledge and skills. The lack of consensus in evaluating continual learning algorithms and the almost exclusive focus on forgetting motivate us to propose a more comprehensive set of implementation independent metrics accounting for several factors we believe have practical implications worth considering in the deployment of real AI systems that learn continually: accuracy or performance over time, backward and forward knowledge transfer, memory overhead as well as computational efficiency. Drawing inspiration from the standard Multi-Attribute Value Theory (MAVT) we further propose to fuse these metrics into a single score for ranking purposes and we evaluate our proposal with five continual learning strategies on the iCIFAR-100 continual learning benchmark.

Paper Structure

This paper contains 6 sections, 9 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction and Related Work
  2. Proposed Metrics for Continual Learning
  3. Experiments and Conclusions
  4. Acknowledgements
  5. Implementation Details
  6. Metrics and benchmark illustration

Figures (2)

  • Figure 1: a) Spider chart: CL metrics per strategy (larger area is better). b) Accuracy per CL strategy computed over the fixed test set as proposed in Lomonaco17Maltoni18.
  • Figure 2: Spider chart with CL metrics showing CL strategies EWC, LWF and SI with their respective lower and upper bound (Naïve and Cumulative resp.) as reference baselines (to properly visualize Fig. \ref{['fig:spider-and-plot']}). The weight configuration for each criterion used is $W_1$ where $w_i =\frac{1}{7}$ for each $w_i \in W$.