Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

F-OAL: Forward-only Online Analytic Learning with Fast Training and Low Memory Footprint in Class Incremental Learning

Huiping Zhuang, Yuchen Liu, Run He, Kai Tong, Ziqian Zeng, Cen Chen, Yi Wang, Lap-Pui Chau

TL;DR

This paper proposes an exemplar-free approach--Forward-only Online Analytic Learning (F-OAL), which does not rely on back-propagation and is forward-only, significantly reducing memory usage and computational time.

Abstract

Online Class Incremental Learning (OCIL) aims to train models incrementally, where data arrive in mini-batches, and previous data are not accessible. A major challenge in OCIL is Catastrophic Forgetting, i.e., the loss of previously learned knowledge. Among existing baselines, replay-based methods show competitive results but requires extra memory for storing exemplars, while exemplar-free (i.e., data need not be stored for replay in production) methods are resource-friendly but often lack accuracy. In this paper, we propose an exemplar-free approach--Forward-only Online Analytic Learning (F-OAL). Unlike traditional methods, F-OAL does not rely on back-propagation and is forward-only, significantly reducing memory usage and computational time. Cooperating with a pre-trained frozen encoder with Feature Fusion, F-OAL only needs to update a linear classifier by recursive least square. This approach simultaneously achieves high accuracy and low resource consumption. Extensive experiments on benchmark datasets demonstrate F-OAL's robust performance in OCIL scenarios. Code is available at https://github.com/liuyuchen-cz/F-OAL.

F-OAL: Forward-only Online Analytic Learning with Fast Training and Low Memory Footprint in Class Incremental Learning

TL;DR

This paper proposes an exemplar-free approach--Forward-only Online Analytic Learning (F-OAL), which does not rely on back-propagation and is forward-only, significantly reducing memory usage and computational time.

Abstract

Online Class Incremental Learning (OCIL) aims to train models incrementally, where data arrive in mini-batches, and previous data are not accessible. A major challenge in OCIL is Catastrophic Forgetting, i.e., the loss of previously learned knowledge. Among existing baselines, replay-based methods show competitive results but requires extra memory for storing exemplars, while exemplar-free (i.e., data need not be stored for replay in production) methods are resource-friendly but often lack accuracy. In this paper, we propose an exemplar-free approach--Forward-only Online Analytic Learning (F-OAL). Unlike traditional methods, F-OAL does not rely on back-propagation and is forward-only, significantly reducing memory usage and computational time. Cooperating with a pre-trained frozen encoder with Feature Fusion, F-OAL only needs to update a linear classifier by recursive least square. This approach simultaneously achieves high accuracy and low resource consumption. Extensive experiments on benchmark datasets demonstrate F-OAL's robust performance in OCIL scenarios. Code is available at https://github.com/liuyuchen-cz/F-OAL.
Paper Structure (21 sections, 37 equations, 4 figures, 6 tables, 1 algorithm)

This paper contains 21 sections, 37 equations, 4 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Proposed Framework
  5. Analytic Solution
  6. Learning Agenda
  7. Complexity Analysis
  8. Overhead Analysis
  9. Experiment
  10. Datasets
  11. Evaluation Metrics
  12. Implementation Details
  13. Result Comparison
  14. Resource Consumption
  15. Countering Recency Bias
  16. ...and 6 more sections

Figures (4)

  • Figure 1: This diagram illustrates the learning agenda of F-OAL. In the encoder , features from each block of the ViT are extracted, summed, and averaged to form a composite feature. This feature is then randomly projected into a higher-dimensional space and normalized using the sigmoid function, serving as the activation for updating the classifier. All parameters in the encoder remain frozen. In the analytic classifier section, we introduce R to retain historical information and update the linear classifier using recursive least squares. This process is forward-only with no gradients.
  • Figure 2: Peak GPU memory usage in GB with 10 batch size on CIFAR-100. Replay-based methods are with 5,000 buffer size. F-OAL has low GPU footprint since it does not require gradients.
  • Figure 3: Visualization of the weights of a linear classifier. The result comes from F-OAL on DTD. Based on hou2019learning, when recency bias happens, L2 norm of current task is significantly larger. In F-OAL, the L2 norm of current task is in a average level.
  • Figure 4: Average accuracy measured in different projection sizes. Due to the time complexity in computing Equation \ref{['eq:rk1']}, higher projection sizes escalate training time while increase little performance. On small datasets, higher projection sizes may even lead to reduction.