Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Conformal Predictive Measure for Assessing Catastrophic Forgetting

Ioannis Pitsiorlas, Nour Jamoussi, Marios Kountouris

TL;DR

This work tackles catastrophic forgetting in continual learning by introducing the Conformal Prediction Confidence Factor (CPCF), an online metric derived from adaptive conformal prediction to quantify forgetting through model uncertainty on previously learned tasks. The methodology splits data into training and calibration sets using a calibration ratio $r$, computes conformal scores $E_i$ and an adjusted quantile $q_α$ during calibration, and forms conformal prediction sets $\\mathcal{C}(x_t)$ during inference to gauge confidence via their average lengths after learning new tasks. Empirical results across MNIST, KMNIST, CIFAR-10, and FashionMNIST show CPCF strongly correlates with the accuracy on previously learned tasks $a_{prev}$, is robust to calibration choices, and becomes more informative under forgetting mitigation with Elastic Weight Consolidation (EWC). The findings establish CPCF as a practical, uncertainty-aware tool for monitoring catastrophic forgetting in dynamic learning environments and suggest extensions to regression and multimodal data to broaden applicability.

Abstract

This work introduces a novel methodology for assessing catastrophic forgetting (CF) in continual learning. We propose a new conformal prediction (CP)-based metric, termed the Conformal Prediction Confidence Factor (CPCF), to quantify and evaluate CF effectively. Our framework leverages adaptive CP to estimate forgetting by monitoring the model's confidence on previously learned tasks. This approach provides a dynamic and practical solution for monitoring and measuring CF of previous tasks as new ones are introduced, offering greater suitability for real-world applications. Experimental results on four benchmark datasets demonstrate a strong correlation between CPCF and the accuracy of previous tasks, validating the reliability and interpretability of the proposed metric. Our results highlight the potential of CPCF as a robust and effective tool for assessing and understanding CF in dynamic learning environments.

A Conformal Predictive Measure for Assessing Catastrophic Forgetting

TL;DR

This work tackles catastrophic forgetting in continual learning by introducing the Conformal Prediction Confidence Factor (CPCF), an online metric derived from adaptive conformal prediction to quantify forgetting through model uncertainty on previously learned tasks. The methodology splits data into training and calibration sets using a calibration ratio , computes conformal scores and an adjusted quantile during calibration, and forms conformal prediction sets during inference to gauge confidence via their average lengths after learning new tasks. Empirical results across MNIST, KMNIST, CIFAR-10, and FashionMNIST show CPCF strongly correlates with the accuracy on previously learned tasks , is robust to calibration choices, and becomes more informative under forgetting mitigation with Elastic Weight Consolidation (EWC). The findings establish CPCF as a practical, uncertainty-aware tool for monitoring catastrophic forgetting in dynamic learning environments and suggest extensions to regression and multimodal data to broaden applicability.

Abstract

This work introduces a novel methodology for assessing catastrophic forgetting (CF) in continual learning. We propose a new conformal prediction (CP)-based metric, termed the Conformal Prediction Confidence Factor (CPCF), to quantify and evaluate CF effectively. Our framework leverages adaptive CP to estimate forgetting by monitoring the model's confidence on previously learned tasks. This approach provides a dynamic and practical solution for monitoring and measuring CF of previous tasks as new ones are introduced, offering greater suitability for real-world applications. Experimental results on four benchmark datasets demonstrate a strong correlation between CPCF and the accuracy of previous tasks, validating the reliability and interpretability of the proposed metric. Our results highlight the potential of CPCF as a robust and effective tool for assessing and understanding CF in dynamic learning environments.
Paper Structure (13 sections, 3 equations, 3 figures, 3 tables, 1 algorithm)

This paper contains 13 sections, 3 equations, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Proposed Methodology
  3. Data Splitting
  4. Calibration Phase
  5. Prediction Phase
  6. Proposed Metric: CPCF
  7. Experimental Results
  8. Datasets Used
  9. Model Architecture
  10. Training Curriculum
  11. Evaluation Metrics
  12. Numerical Results
  13. Discussion and Concluding Remarks

Figures (3)

  • Figure 1: Proposed framework overview for assessing catastrophic forgetting in continual learning via conformal prediction.
  • Figure 2: Impact of learning rate on CF in MNIST. The y-axis represents the accuracy on previously learned tasks.
  • Figure 3: Illustration of Catastrophic Forgetting: Comparison of accuracy on previously learned tasks ($a_{\text{prev}}$) and newly learned tasks ($a_{\text{new}}$) across incremental tasks.