Scalable Forward-Forward Algorithm

TL;DR

The paper tackles the memory and interpretability drawbacks of backpropagation for large CNNs by introducing Scalable Forward-Forward (SFF), a layerwise training paradigm that extends Forward-Forward to modern architectures. It combines a one-pass positive/negative goodness computation with a log-sum-exp margin loss for convolutional layers and a hybrid training scheme that allows backpropagation within model blocks while remaining forward-only between blocks. Key contributions include the auxiliary 1x1 convolutional layers for class-specific goodness, a margin-based objective, and demonstrated effectiveness on ResNet18 and MobileNetV3 across CIFAR-10/100 and Imagenette, including small-data and transfer-learning regimes. The approach offers memory efficiency, modularity, and competitive performance, making it a practical replacement for backpropagation in large-scale CNNs, with code available for reproducibility.

Abstract

We propose a scalable Forward-Forward (FF) algorithm that eliminates the need for backpropagation by training each layer separately. Unlike backpropagation, FF avoids backward gradients and can be more modular and memory efficient, making it appealing for large networks. We extend FF to modern convolutional architectures, such as MobileNetV3 and ResNet18, by introducing a new way to compute losses for convolutional layers. Experiments show that our method achieves performance comparable to standard backpropagation. Furthermore, when we divide the network into blocks, such as the residual blocks in ResNet, and apply backpropagation only within each block, but not across blocks, our hybrid design tends to outperform backpropagation baselines while maintaining a similar training speed. Finally, we present experiments on small datasets and transfer learning that confirm the adaptability of our method.