A Basic Evaluation of Neural Networks Trained with the Error Diffusion Learning Algorithm

TL;DR

This paper formulates Kaneko's Error Diffusion Learning Algorithm (EDLA) as a biologically plausible alternative to backpropagation and evaluates it across parity checks, regression, and image-classification benchmarks. EDLA diffuses a single global error signal $d$ through paired positive/negative sublayers, avoiding layer-wise backprop while aiming to minimize the loss $E(W)$; it also introduces RMS normalization to stabilize training, particularly for ReLU activations. Across tasks, EDLA shows competitive performance in shallow networks but exhibits a notable performance gap relative to backpropagation as task complexity and depth increase, with stability improvements from the proposed initialization and normalization techniques. The work highlights the potential of EDLA for neuromorphic and neuroscience-informed learning, provides extensive empirical evidence, and shares open-source code for reproducibility.

Abstract

This paper presents a comprehensive formulation of Kaneko's Error Diffusion Learning Algorithm (EDLA) and evaluates its effectiveness across parity check, regression, and image classification tasks. EDLA is a biologically inspired learning algorithm that provides an alternative to conventional backpropagation for training artificial neural networks. EDLA employs a single global error signal that diffuses across networks composed of paired positive and negative sublayers, eliminating traditional layer-wise error backpropagation. This study evaluates EDLA's effectiveness using benchmark tasks, such as parity check, regression, and image classification, by systematically varying the neuron count, network depth, and learning rates to assess its performance comprehensively. The experimental results demonstrate that EDLA achieves consistently high accuracy across multiple benchmarks, highlighting its effectiveness as a learning algorithm for neural networks. The choice of learning rate, neuron count, and network depth significantly influences EDLA's efficiency and convergence speed. Analysis of internal network representations reveals meaningful feature extraction capabilities, and the network's overall performance is found to be competitive with networks trained via conventional backpropagation, especially in shallow architectures. This study introduces EDLA, a biologically plausible alternative to traditional backpropagation previously underrecognized due to language barriers. By reformulating EDLA, systematically evaluating its performance, and presenting empirical evidence of its effectiveness, this study increases the visibility and accessibility of EDLA and contributes to biologically inspired training methodologies.

Figures (12)

• Figure 1: Schematic representation of the Error Diffusion Learning Algorithm (EDLA) network architecture. The network consists of multiple layers of positive neurons (white circles) and negative neurons (gray circles). The red and blue small circles at the connections indicate excitatory and inhibitory synapses, respectively.
• Figure 2: Performance of EDLA with the sigmoid AF across various learning rates on the parity check task. The left column depicts the accuracy of the EDLA network with the sigmoid AF as a function of neurons in a single hidden layer for different learning rates (LR = 0.1, 1.0, 10.0). The accuracies are averaged from 19900 to 20000 epochs. The right column shows the number of epochs required to reach an accuracy of 0.9 as a function of the number of neurons in a hidden layer for the same learning rates. Each row of plots corresponds to a different bit length (n_bit, ranging from 2 to 5), for the parity check task. The results are averaged over ten independent trials.
• Figure 3: Accuracy and convergence characteristics of the EDLA network with the sigmoid AF across various network depths (1, 2, 4, and 8 hidden layers) at a fixed learning rate of 1.0. The left column shows the average classification accuracy as a function of the neurons per hidden layer. The accuracies are averaged from 19900 to 20000 epochs. The right column presents the number of epochs required to reach an accuracy of 0.9 as a function of the number of neurons per hidden layer. Each row of plots corresponds to different bit lengths (n_bit, ranging from 2 to 5). The results are averaged over ten independent trials.
• Figure 4: Performance of EDLA with the ReLU AF across various learning rates in the parity check task. The left column shows the accuracy of the EDLA network as a function of neurons in a single hidden layer for different learning rates (LR = 0.01, 0.1, 1.0). The accuracies are averaged from 19900 to 20000 epochs. The right column shows the number of epochs required to achieve an accuracy of 0.9 against the number of neurons in the hidden layer for the same learning rates. Each row of plots corresponds to a different bit length (n_bit), ranging from 2 to 5, for the parity check task. Missing data points in the left column indicate that the accuracy is excessively low. Missing data points in the right column indicate that the accuracy does not reach 0.9. The results are averaged over ten independent trials.
• Figure 5: Accuracy and convergence characteristics of the EDLA network with ReLU AF across various network depths (1, 2, 4, and 8 hidden layers), using a fixed learning rate of 0.1. The left column shows average classification accuracy as a function of neurons per hidden layer. The accuracies are averaged from 19900 to 20000 epochs. The right column presents the number of epochs required to reach an accuracy of 0.9 as a function of the number of neurons per hidden layer. Each row of plots corresponds to different bit lengths (n_bit), ranging from 2 to 5. The results are averaged over ten independent trials.