Function regression using the forward forward training and inferring paradigm

TL;DR

This work extends Forward-Forward learning from classification to function regression, reframing regression as in-tol/out-tol classification and training each layer with a goodness-based objective. It uses a cosine-similarity goodness with a fixed per-layer vector and a layer-wise loss to maximize positive-versus-negative discrimination, with inference aggregating layer goodness across candidate labels. The method is demonstrated on 1D, 2D, and 3D functions, revealing sensible mean predictions and reduced uncertainty with more data and epochs, while noting a goodness-inversion near tol regions and occasional difficulty with highly periodic components. Preliminary explorations of Kolmogorov Arnold Networks and Deep Physical Neural Networks are reported, alongside a comparison to backpropagation on standard hardware, which is faster but may not capture the energy-efficiency advantages of FF in analog contexts.

Abstract

Function regression/approximation is a fundamental application of machine learning. Neural networks (NNs) can be easily trained for function regression using a sufficient number of neurons and epochs. The forward-forward learning algorithm is a novel approach for training neural networks without backpropagation, and is well suited for implementation in neuromorphic computing and physical analogs for neural networks. To the best of the authors' knowledge, the Forward Forward paradigm of training and inferencing NNs is currently only restricted to classification tasks. This paper introduces a new methodology for approximating functions (function regression) using the Forward-Forward algorithm. Furthermore, the paper evaluates the developed methodology on univariate and multivariate functions, and provides preliminary studies of extending the proposed Forward-Forward regression to Kolmogorov Arnold Networks, and Deep Physical Neural Networks.

Figures (18)

• Figure 1: (a) Schematic diagram of neural network which can be trained using forward forward algorithm. Note that the last layer is also the same as a hidden layer, i.e., there is no output from the final layer. (b) Arbitrary vectors used to optimize the difference between positive and negative goodness.
• Figure 2: Schematic diagram for training of a FF NN, with the red crosses indicating the training data-points, and colored circles indicating the trial points with green corresponding to label 1.0 and red corresponding to label 0.0. On the left, the positive data has the trial points labeled correctly, while the right figure shows the negative data with incorrect labeling.
• Figure 3: Schematic diagram of prediction phase while inferencing from the trained forward forward neural network. Both labels-- in-tol (1.0) and out-tol (0.0) are applied to all the trial points. The label yielding the higher goodness value would ideally be chosen as the correct label for the trial point (see \ref{['subsubsec:goodnessinvert']}).
• Figure 4: Results of FF-regrssion on 1D functions-- (a) $f_1(x)$, (b) $f_2(x)$ and (c) $f_3(x)$, with the red crosses indicating the training data points, blue line indicating the mean predicted value, while the shaded area denotes the 95% confidence interval.
• Figure 5: FF-Regression results for the 2D functions-- (a) $f_4(x_1,x_2)$ and (b) $f_5(x_1,x_2)$, with the yellow surface indicating the actual function output and the green surface indicating the mean predicted function values. The training datapoints and confidence bounds are omitted for clarity.