Error Broadcast and Decorrelation as a Potential Artificial and Natural Learning Mechanism

TL;DR

The paper introduces Error Broadcast and Decorrelation (EBD), a principled framework for credit assignment that broadcasts output errors to all layers and minimizes layerwise decorrelation with the error, grounded in nonlinear MMSE orthogonality. By constructing layer-specific losses that enforce $\mathbb{E}[\mathbf{g}(\mathbf{x})\bm{\epsilon}^T]=\mathbf{0}$, EBD yields localized, potentially parallel updates and a natural emergence of three-factor learning rules. It extends to biologically realistic networks, including CorInfoMax-EBD variants, and demonstrates competitive performance against other error-broadcast methods on MNIST and CIFAR benchmarks, with deep CorInfoMax-EBD networks showing notable depth scalability. The approach offers a principled alternative to backpropagation that aligns with biological plausibility and hardware-efficient computation, while acknowledging open questions around scalability, convergence in finite-width regimes, and computational overhead. Overall, EBD provides a new lens on credit assignment that unifies estimation-theoretic orthogonality with practical, biologically plausible learning dynamics.

Abstract

We introduce Error Broadcast and Decorrelation (EBD), a novel learning framework for neural networks that addresses credit assignment by directly broadcasting output errors to individual layers, circumventing weight transport of backpropagation. EBD is rigorously grounded in the stochastic orthogonality property of Minimum Mean Square Error estimators. This fundamental principle states that the error of an optimal estimator is orthogonal to functions of the input. Guided by this insight, EBD defines layerwise loss functions that directly penalize correlations between layer activations and output errors, thereby establishing a principled foundation for error broadcasting. This theoretically sound mechanism naturally leads to the experimentally observed three-factor learning rule and integrates with biologically plausible frameworks to enhance performance and plausibility. Numerical experiments demonstrate EBD's competitive or better performance against other error-broadcast methods on benchmark datasets. Our findings establish EBD as an efficient, biologically plausible, and principled alternative for neural network training. The implementation is available at: https://github.com/meterdogan07/error-broadcast-decorrelation.

Figures (12)

• Figure 1: Comparison of error feedback mechanisms and correlation dynamics in multilayer perceptrons. (a) Backpropagation (BP) transmits errors sequentially through symmetric backward paths. (b) Error Broadcast and Decorrelation (EBD) broadcasts output errors to all layers using error--activation cross-correlations. (c) Average absolute correlation between layer activations and the output error during BP training on CIFAR-10 with MSE loss, illustrating its decline over epochs (see Appendix \ref{['sec:correlation_calculation']}).
• Figure 2: Error–broadcast learning as a three‑factor synaptic update. The presynaptic firing rate $h^{(k-1)}_j$ (green, left) projects onto the postsynaptic neuron $h^{(k)}_i$ (blue, centre) through the synapse $W^{(k)}_{ij}$. A layer‑specific broadcast of the output error $e\!\to\!q^{(k)}_i$ (yellow, right) provides the modulatory third factor that gates plasticity. Together, presynaptic activity, postsynaptic non‑linear derivatives $g^{\prime(k)}_i f^{\prime(k)}$ (blue rectangle), and the modulatory signal form the product that drives the EBD weight change $\Delta W^{(k)}_{ij}$ displayed underneath the circuit.
• Figure 3: Cosine similarity between EBD updates and backpropagation gradients in a 3-layer MLP trained on CIFAR-10. Alignment is consistently positive and improves during training.
• Figure 4: Cosine similarity between EBD updates and backpropagation gradients in a locally connected network on CIFAR-10. Positive alignment indicates directional consistency between EBD and BP.
• Figure 5: Cosine similarity between the full decorrelation gradient (including $(\Delta W_2^{(k)}, \Delta b_2^{(k)})$) and the truncated EBD update (only $(\Delta W_1^{(k)}, \Delta b_1^{(k)})$). Positive similarity confirms that the truncated update remains a valid descent direction.

Theorems & Definitions (8)

• Lemma A.1
• Lemma A.2
• proof
• proof
• Theorem B.1: Nonlinear MMSE Estimation and Orthogonality Condition
• proof
• Theorem B.2: Convergence to MMSE for Estimators Orthogonal to a Dense Basis of First-Layer Activations
• proof