Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Contrastive Symmetric Forward-Forward Algorithm (SFFA) for Continual Learning Tasks

Erik B. Terres-Escudero, Javier Del Ser, Pablo Garcia Bringas

TL;DR

This paper introduces the Symmetric Forward-Forward Algorithm (SFFA), a symmetric extension of the Forward-Forward Algorithm (FFA) designed to address asymmetries in FFA's loss landscape and improve generalization for continual learning. By partitioning each layer into positive and negative neuron groups and defining a balanced probability-based goodness, SFFA achieves more stable, layer-wise updates and naturally supports sparse latent representations favorable for CL. The authors demonstrate that SFFA improves generalization over FFA, achieves competitive performance with backpropagation on several image datasets, and can be integrated with standard CL methods across Class IL, Domain IL, and Task IL, albeit with limitations for Task IL due to weight-sharing constraints. The work identifies a promising direction for forward-only learning in continual learning settings and outlines future avenues, including architectural adaptations and convolutional enhancements, to broaden applicability and performance.

Abstract

The so-called Forward-Forward Algorithm (FFA) has recently gained momentum as an alternative to the conventional back-propagation algorithm for neural network learning, yielding competitive performance across various modeling tasks. By replacing the backward pass of gradient back-propagation with two contrastive forward passes, the FFA avoids several shortcomings undergone by its predecessor (e.g., vanishing/exploding gradient) by enabling layer-wise training heuristics. In classification tasks, this contrastive method has been proven to effectively create a latent sparse representation of the input data, ultimately favoring discriminability. However, FFA exhibits an inherent asymmetric gradient behavior due to an imbalanced loss function between positive and negative data, adversely impacting on the model's generalization capabilities and leading to an accuracy degradation. To address this issue, this work proposes the Symmetric Forward-Forward Algorithm (SFFA), a novel modification of the original FFA which partitions each layer into positive and negative neurons. This allows the local fitness function to be defined as the ratio between the activation of positive neurons and the overall layer activity, resulting in a symmetric loss landscape during the training phase. To evaluate the enhanced convergence of our method, we conduct several experiments using multiple image classification benchmarks, comparing the accuracy of models trained with SFFA to those trained with its FFA counterpart. As a byproduct of this reformulation, we explore the advantages of using a layer-wise training algorithm for Continual Learning (CL) tasks. The specialization of neurons and the sparsity of their activations induced by layer-wise training algorithms enable efficient CL strategies that incorporate new knowledge (classes) into the neural network, while preventing catastrophic forgetting of previously...

A Contrastive Symmetric Forward-Forward Algorithm (SFFA) for Continual Learning Tasks

TL;DR

This paper introduces the Symmetric Forward-Forward Algorithm (SFFA), a symmetric extension of the Forward-Forward Algorithm (FFA) designed to address asymmetries in FFA's loss landscape and improve generalization for continual learning. By partitioning each layer into positive and negative neuron groups and defining a balanced probability-based goodness, SFFA achieves more stable, layer-wise updates and naturally supports sparse latent representations favorable for CL. The authors demonstrate that SFFA improves generalization over FFA, achieves competitive performance with backpropagation on several image datasets, and can be integrated with standard CL methods across Class IL, Domain IL, and Task IL, albeit with limitations for Task IL due to weight-sharing constraints. The work identifies a promising direction for forward-only learning in continual learning settings and outlines future avenues, including architectural adaptations and convolutional enhancements, to broaden applicability and performance.

Abstract

The so-called Forward-Forward Algorithm (FFA) has recently gained momentum as an alternative to the conventional back-propagation algorithm for neural network learning, yielding competitive performance across various modeling tasks. By replacing the backward pass of gradient back-propagation with two contrastive forward passes, the FFA avoids several shortcomings undergone by its predecessor (e.g., vanishing/exploding gradient) by enabling layer-wise training heuristics. In classification tasks, this contrastive method has been proven to effectively create a latent sparse representation of the input data, ultimately favoring discriminability. However, FFA exhibits an inherent asymmetric gradient behavior due to an imbalanced loss function between positive and negative data, adversely impacting on the model's generalization capabilities and leading to an accuracy degradation. To address this issue, this work proposes the Symmetric Forward-Forward Algorithm (SFFA), a novel modification of the original FFA which partitions each layer into positive and negative neurons. This allows the local fitness function to be defined as the ratio between the activation of positive neurons and the overall layer activity, resulting in a symmetric loss landscape during the training phase. To evaluate the enhanced convergence of our method, we conduct several experiments using multiple image classification benchmarks, comparing the accuracy of models trained with SFFA to those trained with its FFA counterpart. As a byproduct of this reformulation, we explore the advantages of using a layer-wise training algorithm for Continual Learning (CL) tasks. The specialization of neurons and the sparsity of their activations induced by layer-wise training algorithms enable efficient CL strategies that incorporate new knowledge (classes) into the neural network, while preventing catastrophic forgetting of previously...
Paper Structure (25 sections, 16 equations, 8 figures, 6 tables, 2 algorithms)

This paper contains 25 sections, 16 equations, 8 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Continual Learning
  4. The Forward-Forward Algorithm
  5. Proposed Symmetric Forward-Forward Algorithm (SFFA)
  6. SFFA for Continual Learning
  7. Motivation
  8. Adaptation
  9. Limitations
  10. Experimental Setup
  11. Results and Discussions
  12. RQ1: Does the removal of the asymmetry in the SFFA goodness function enhance its generalization capabilities?
  13. RQ2: Can SFFA be used with conventional CL methods to solve IL scenarios?
  14. Conclusions and Future Work
  15. Algorithm to Adapt FFA to CL
  16. ...and 10 more sections

Figures (8)

  • Figure 1: Diagram illustrating the architecture of a) Backpropagation (BP), b) Forward-Forward Algorithm (FFA) hinton2022forward, and c) Symmetric Forward-Forward Algorithm (SFFA). Each architecture highlights the input forward path (black arrows), and the error propagation path (blue arrows). Additionally, we describe the update mechanism of FFA and SFFA, and also show the probability function of SFFA.
  • Figure 2: Average classification accuracy of the gradient back-propagation algorithm and SFFA for different IL scenarios: Class IL (first column), Domain IL (center column) and Task IL (right column). Each point captures the average accuracy of the model (over 10 runs) on all evaluation data from tasks that the model has been previously trained on.
  • Figure 3: Visualization of the probability function of FFA (left) and the SFFA (right). The x-axis in the plot corresponding to FFA contains the total activity of the layer, whereas its y-axis specifies the threshold. The x-axis of the SFFA plot denotes the activity of the positive neurons, whereas the y-axis depicts the activity of the negative neurons.
  • Figure 4: Distribution of neurons sharing more than $1\%$ of the latent vectors activation. The x-axis stands for the amount of active neurons, while the y-axis shows the total number of latent vectors with an specific number of high-activity neurons.
  • Figure 5: Aggregated latent activations of the first layer over each test dataset for SFFA (first row) and FFA (second layer). The latent activation vector has been reshaped to a 28 by 50 rectangle for the sake of readability. The color of each neuron displays its activity, with darker colors representing less activity across the dataset at hand.
  • ...and 3 more figures