Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Task Confusion and Catastrophic Forgetting in Class-Incremental Learning: A Mathematical Framework for Discriminative and Generative Modelings

Milad Khademi Nori, Il-Min Kim

TL;DR

A novel mathematical framework is presented and the Infeasibility Theorem is established, demonstrating that generative modeling can achieve optimal class-IL by overcoming task confusion, and adopting generative modeling, either for generative replay or direct classification, is essential for optimal class-IL.

Abstract

In class-incremental learning (class-IL), models must classify all previously seen classes at test time without task-IDs, leading to task confusion. Despite being a key challenge, task confusion lacks a theoretical understanding. We present a novel mathematical framework for class-IL and prove the Infeasibility Theorem, showing optimal class-IL is impossible with discriminative modeling due to task confusion. However, we establish the Feasibility Theorem, demonstrating that generative modeling can achieve optimal class-IL by overcoming task confusion. We then assess popular class-IL strategies, including regularization, bias-correction, replay, and generative classifier, using our framework. Our analysis suggests that adopting generative modeling, either for generative replay or direct classification (generative classifier), is essential for optimal class-IL.

Task Confusion and Catastrophic Forgetting in Class-Incremental Learning: A Mathematical Framework for Discriminative and Generative Modelings

TL;DR

A novel mathematical framework is presented and the Infeasibility Theorem is established, demonstrating that generative modeling can achieve optimal class-IL by overcoming task confusion, and adopting generative modeling, either for generative replay or direct classification, is essential for optimal class-IL.

Abstract

In class-incremental learning (class-IL), models must classify all previously seen classes at test time without task-IDs, leading to task confusion. Despite being a key challenge, task confusion lacks a theoretical understanding. We present a novel mathematical framework for class-IL and prove the Infeasibility Theorem, showing optimal class-IL is impossible with discriminative modeling due to task confusion. However, we establish the Feasibility Theorem, demonstrating that generative modeling can achieve optimal class-IL by overcoming task confusion. We then assess popular class-IL strategies, including regularization, bias-correction, replay, and generative classifier, using our framework. Our analysis suggests that adopting generative modeling, either for generative replay or direct classification (generative classifier), is essential for optimal class-IL.

Paper Structure

This paper contains 17 sections, 48 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Mathematical framework for Class-IL
  3. Discriminative modeling
  4. Generative modeling
  5. Optimality analysis of Class-IL strategies
  6. Regularization strategies
  7. Distillation strategies
  8. Bias-correction strategies
  9. Generative replay
  10. Generative classifier
  11. Big picture and conclusion
  12. Complexity analysis
  13. Related Work
  14. Regularization strategy
  15. Bias-correction strategy
  16. ...and 2 more sections

Figures (5)

  • Figure 1: Task Confusion in Discriminative and Generative Modeling.
  • Figure 2: Task Confusion in Discriminative and Generative Modeling.
  • Figure 3: In all the six figures, for the task-IL scenario, the schemes merely face CF (because they are given with the task-ID), and thus, they perform favorably. In the class-IL scenario, however, the models need to discriminate between different tasks, and they usually fail; this is expected due to not minimizing the inter-task block losses.
  • Figure 4: Generative classifiers like SLDA and GenC mitigate TC and CF (CIFAR-10).
  • Figure 5: Figure F.1: In this figure, Synaptic Intelligence (SI) zenke2017continual with $\lambda=1$ is adopted. It is clear that SI is almost effective at mitigating CF; however, ineffective for TC.