FFCL: Forward-Forward Net with Cortical Loops, Training and Inference on Edge Without Backpropagation

TL;DR

The paper tackles memory and compute bottlenecks of backpropagation for edge deployment by building on Forward-Forward Learning (FFL). It introduces two variants: Forward-Forward Direct Labeling (FFDL), which feeds labels directly to every layer, and Forward-Forward Cortical Loops (FFCL), which adds bidirectional, cortical-like feedback via unrolled networks with weight sharing. Across MNIST, Fashion-MNIST, and CIFAR-10, these methods achieve higher per-layer and final accuracies while substantially reducing inference FLOPs relative to FFL. The results indicate strong potential for memory-efficient, on-device learning with minimal activation-storing requirements, enabling practical edge deployment of forward-forward schemes.

Abstract

The Forward-Forward Learning (FFL) algorithm is a recently proposed solution for training neural networks without needing memory-intensive backpropagation. During training, labels accompany input data, classifying them as positive or negative inputs. Each layer learns its response to these inputs independently. In this study, we enhance the FFL with the following contributions: 1) We optimize label processing by segregating label and feature forwarding between layers, enhancing learning performance. 2) By revising label integration, we enhance the inference process, reduce computational complexity, and improve performance. 3) We introduce feedback loops akin to cortical loops in the brain, where information cycles through and returns to earlier neurons, enabling layers to combine complex features from previous layers with lower-level features, enhancing learning efficiency.

Figures (4)

• Figure 1: hinton2022forward embeds labels in MNIST's black border, altering the first 10 pixels for class representation: '1' for the true class in positive samples and '1' in a random other class for negative samples, with the rest as '0'
• Figure 2: Forward-Forward Algorithm Setup: Transforms MNIST images to a 784-unit input layer. For class representation, the initial N pixels are adjusted: '1' for the correct class in positive samples and '1' in a non-class index for negative samples, particularly in the first 10 pixels for MNIST. Layers are trained sequentially.
• Figure 3: In our revised FFL we directly integrate class labels into the structure, maintaining original image integrity. Training links input and labels to initial layer neurons, enabling specialized computations. The following layers focus on weighted images and bias from the prior layer, optimizing processing.
• Figure 4: Architecture of Forward-Forward Net with Cortical Loops. To train the model effectively using existing training engines, the model is unrolled. The number of times, the model is unrolled decides the degree to which feedback information propagates in the system. For example, with a 3 (N) layer network, each layer feedback could reach 2 (N-1) previous layers.