Implicit Regularization in Feedback Alignment Learning Mechanisms for Neural Networks

TL;DR

The paper addresses the gap in understanding how Feedback Alignment (FA) can learn effectively while remaining biologically plausible. It introduces a modular framework that decomposes learning Dynamics into alignment, gradient, and FA components, and proves a conservation law that enforces alignment at the neuron level, enabling an alignment-dominance-based convergence analysis. Theoretical results show that, under certain data separability conditions and initialization schemes, FA (including sign-FA and adaFA) converges, with explicit exponential or generalized decay of the loss, and that alignment quality correlates with improved multi-class performance. Empirically, the framework is validated on MNIST, CIFAR-100, and Tiny-ImageNet, demonstrating alignment conservation, enhanced performance with better alignment, and insights into benign overfitting in FA-based learning. These findings advance interpretability of bio-plausible learning and provide practical guidance for designing FA-based algorithms with improved convergence and generalization in complex tasks.

Abstract

Feedback Alignment (FA) methods are biologically inspired local learning rules for training neural networks with reduced communication between layers. While FA has potential applications in distributed and privacy-aware ML, limitations in multi-class classification and lack of theoretical understanding of the alignment mechanism have constrained its impact. This study introduces a unified framework elucidating the operational principles behind alignment in FA. Our key contributions include: (1) a novel conservation law linking changes in synaptic weights to implicit regularization that maintains alignment with the gradient, with support from experiments, (2) sufficient conditions for convergence based on the concept of alignment dominance, and (3) empirical analysis showing better alignment can enhance FA performance on complex multi-class tasks. Overall, these theoretical and practical advancements improve interpretability of bio-plausible learning rules and provide groundwork for developing enhanced FA algorithms.

Figures (7)

• Figure 1: Implicit Regularization of Alignment. We show an idealization of a neuron. The solid arrows indicate synaptic connections. In matrix form, $W_i[k,j]$ indicates a connection between the $k^{\text{th}}$ neuron in the $i^{\text{th}}$ layer to the $j^{\text{th}}$ neuron in the next layer. The matrix $B_{i+1}$ indicates a feedback connection. We establish a hard equality constraint that relates the alignment between incoming, outgoing, and feedback weights. See Theorem \ref{['conservation_law']} for more details.
• Figure 2: Conservation of Alignment. We measure the "potential" $\langle W_{2}^{(t)}, B_{2} \rangle - \frac{1}{2} \Vert W_1(t) \Vert_F^2$ during training on MNIST with varying width. We normalize by the value at initialization and so Theorem \ref{['conservation_law']} predicts this measure to be constant and equal to one. For gradient descent there is a similar result; change of each layer's squared norm is equal du2018algorithmic. See Section \ref{['empirical']} for more details.
• Figure 3: Alignment During Training: We plot the cosine of the angle between forward ($W_2^{(t)}$) and feedback ($B_2$) weights during training on Noisy MNIST with 20% label noise across varying network widths. See Section \ref{['empirical']} for more details.
• Figure 4: Benign Overfitting: We plot the test-loss curve as a function of width for networks trained on noisy MNIST with 20% label noise. Despite fitting the noise all methods are able to generalize.
• Figure 5: 10-Class Alignment: We plot the cosine of the angle between forward and backward weights during training on $10$-class subsets of CIFAR-100.

Theorems & Definitions (19)

• Definition 4.1
• Definition 4.2
• Theorem 5.1: Convergence under Alignment Dominance
• Theorem 5.2
• Proposition 5.2
• proof
• Lemma 5.2
• proof
• Definition 5.2
• Definition 5.2