The Forward-Forward Algorithm: Characterizing Training Behavior

TL;DR

This work analyzes the training dynamics of Forward-Forward networks, an alternative to backpropagation that uses two forward passes and layer-local goodness objectives. By varying network depth, width, and training epochs on MNIST, it demonstrates that deeper layers exhibit delayed accuracy gains while shallower layers' performance correlates more strongly with overall accuracy. The study introduces and tests hypotheses about depth-related delays and layer-to-global accuracy correlations, finding that layer depth conditions these relationships and that correlations weaken with depth or increased layer width. These insights contribute to a mechanistic understanding of Forward-Forward behavior and suggest practical directions for layer-wise network design and iterative construction, though broader validation across datasets is needed.

Abstract

The Forward-Forward algorithm is an alternative learning method which consists of two forward passes rather than a forward and backward pass employed by backpropagation. Forward-Forward networks employ layer local loss functions which are optimized based on the layer activation for each forward pass rather than a single global objective function. This work explores the dynamics of model and layer accuracy changes in Forward-Forward networks as training progresses in pursuit of a mechanistic understanding of their internal behavior. Treatments to various system characteristics are applied to investigate changes in layer and overall model accuracy as training progresses, how accuracy is impacted by layer depth, and how strongly individual layer accuracy is correlated with overall model accuracy. The empirical results presented suggest that layers deeper within Forward-Forward networks experience a delay in accuracy improvement relative to shallower layers and that shallower layer accuracy is strongly correlated with overall model accuracy.

Figures (5)

• Figure 1: A simplified diagram of a Forward-Forward network. Weights in each layer are adjusted to increase the goodness response of a layer when presented with a positive sample and decrease the goodness response of a layer when presented with a negative sample. Lowe
• Figure 2: Layer accuracy as training progresses. Each subplot represents a different experimental configuration; subplot column is defined by the dimension of each individual hidden layer while subplot row indicates the number of hidden layers. For each individual subplot the x-axis represents the training epoch at which measurement occurred and the y-axis represents the accuracy measured. Each data point represents the average accuracy across 10 repetitions with different initialization seeds. Trace color represents relative layer depth. Overall model accuracy is indicated as a dashed line
• Figure 3: Continuous error bars for a selection of layers for a Forward-Forward model with 32 layers and 1000 activations per layer. The error bars represent one standard deviation from the mean based on 30 repetitions with random initialization seeds.
• Figure 4: Epoch at which a layer achieves individual target accuracy of 0.7. Each trace represents a specific model configuration defined by the number of hidden layers, denoted by marker style, and the dimension of each hidden layer, denoted by line style. The points on each trace are determined based on the epoch at which a layer at a specific depth achieves the target layer accuracy of 0.7. The x-axis indicates the layer depth, while the y-axis indicates the epoch at which the target was reached. In relation to Figure \ref{['fig:accuracy_subplot']}, this figure represents the epoch value of intersection points of a horizontal line at 0.7 accuracy with each individual layer accuracy trace.
• Figure 5: Correlation of individual layer accuracy with overall network accuracy. \ref{['fig:pearson']} illustrates the Pearson correlation coefficient at each layer and provides a measure of any linear relationship between individual layer accuracy and overall network accuracy. \ref{['fig:spearman']} illustrates Spearman's rank correlation coefficient at each layer and provides a measure of any monotonic relationship between individual layer accuracy and overall network accuracy. Both Pearson and Spearman's rank correlation coefficient may vary between -1 and 1; the sign of the coefficient represents the direction of the relationship while the magnitude of the coefficient represents the strength of the relationship. The correlation for each layer is based on all accuracy measurements across every epoch for the layer and for each experimental repetition with varied initialization seeds.