Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convolutional Channel-wise Competitive Learning for the Forward-Forward Algorithm

Andreas Papachristodoulou, Christos Kyrkou, Stelios Timotheou, Theocharis Theocharides

TL;DR

The main ideas of FF are taken and improved by leveraging channel-wise competitive learning in the context of convolutional neural networks for image classification tasks and a layer-wise loss function is introduced that promotes competitive learning and eliminates the need for negative data construction.

Abstract

The Forward-Forward (FF) Algorithm has been recently proposed to alleviate the issues of backpropagation (BP) commonly used to train deep neural networks. However, its current formulation exhibits limitations such as the generation of negative data, slower convergence, and inadequate performance on complex tasks. In this paper, we take the main ideas of FF and improve them by leveraging channel-wise competitive learning in the context of convolutional neural networks for image classification tasks. A layer-wise loss function is introduced that promotes competitive learning and eliminates the need for negative data construction. To enhance both the learning of compositional features and feature space partitioning, a channel-wise feature separator and extractor block is proposed that complements the competitive learning process. Our method outperforms recent FF-based models on image classification tasks, achieving testing errors of 0.58%, 7.69%, 21.89%, and 48.77% on MNIST, Fashion-MNIST, CIFAR-10 and CIFAR-100 respectively. Our approach bridges the performance gap between FF learning and BP methods, indicating the potential of our proposed approach to learn useful representations in a layer-wise modular fashion, enabling more efficient and flexible learning.

Convolutional Channel-wise Competitive Learning for the Forward-Forward Algorithm

TL;DR

The main ideas of FF are taken and improved by leveraging channel-wise competitive learning in the context of convolutional neural networks for image classification tasks and a layer-wise loss function is introduced that promotes competitive learning and eliminates the need for negative data construction.

Abstract

The Forward-Forward (FF) Algorithm has been recently proposed to alleviate the issues of backpropagation (BP) commonly used to train deep neural networks. However, its current formulation exhibits limitations such as the generation of negative data, slower convergence, and inadequate performance on complex tasks. In this paper, we take the main ideas of FF and improve them by leveraging channel-wise competitive learning in the context of convolutional neural networks for image classification tasks. A layer-wise loss function is introduced that promotes competitive learning and eliminates the need for negative data construction. To enhance both the learning of compositional features and feature space partitioning, a channel-wise feature separator and extractor block is proposed that complements the competitive learning process. Our method outperforms recent FF-based models on image classification tasks, achieving testing errors of 0.58%, 7.69%, 21.89%, and 48.77% on MNIST, Fashion-MNIST, CIFAR-10 and CIFAR-100 respectively. Our approach bridges the performance gap between FF learning and BP methods, indicating the potential of our proposed approach to learn useful representations in a layer-wise modular fashion, enabling more efficient and flexible learning.
Paper Structure (25 sections, 7 equations, 7 figures, 6 tables, 1 algorithm)

This paper contains 25 sections, 7 equations, 7 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Proposed Approach
  4. Channel-wise Competitive Learning
  5. Formulating Convolutional Goodness.
  6. PvN Loss Function.
  7. CwC Loss Function.
  8. Network Architecture
  9. CFSE block.
  10. Channel-wise Grouped Convolutions.
  11. Predictors.
  12. Interleaved Layer Training (ILT) Strategy
  13. Experimental Design and Results
  14. Ablation Study
  15. Impact of Loss Function.
  16. ...and 10 more sections

Figures (7)

  • Figure 1: Top: Network Architecture, Components of CFSE Block, Standard Convolutional (Conv) Layer, Grouped Convolutional (Group - Conv) Layer; Bottom: Non-separable Convolution, Channel-wise Group - Conv, CwC Loss Function.
  • Figure 2: Feature Map Visualization generated by the CFSE_CwC’s CNN layers on MNIST inputs (Left:7; Right:4). Rows correspond to layers (Block1: Conv, GroupConv; Block2: Conv, GroupConv), and columns to the subsets of channels tied to each class. The activations that correspond to the true class for both examples exhibit higher values on the last two layers.
  • Figure 3: Feature maps for Target-class: 3
  • Figure 4: Feature maps for Target-class: 6
  • Figure 5: Feature maps for Target-class: 9
  • ...and 2 more figures