Guiding Sparse Neural Networks with Neurobiological Principles to Elicit Biologically Plausible Representations

TL;DR

This study introduces a biologically inspired learning rule that naturally integrates neurobiological principles, including sparsity, lognormal weight distributions, and adherence to Dale's law, without requiring explicit enforcement, and shows superior generalization in few-shot learning scenarios.

Abstract

While deep neural networks (DNNs) have achieved remarkable performance in tasks such as image recognition, they often struggle with generalization, learning from few examples, and continuous adaptation - abilities inherent in biological neural systems. These challenges arise due to DNNs' failure to emulate the efficient, adaptive learning mechanisms of biological networks. To address these issues, we explore the integration of neurobiologically inspired assumptions in neural network learning. This study introduces a biologically inspired learning rule that naturally integrates neurobiological principles, including sparsity, lognormal weight distributions, and adherence to Dale's law, without requiring explicit enforcement. By aligning with these core neurobiological principles, our model enhances robustness against adversarial attacks and demonstrates superior generalization, particularly in few-shot learning scenarios. Notably, integrating these constraints leads to the emergence of biologically plausible neural representations, underscoring the efficacy of incorporating neurobiological assumptions into neural network design. Preliminary results suggest that this approach could extend from feature-specific to task-specific encoding, potentially offering insights into neural resource allocation for complex tasks.

Figures (4)

• Figure 1: Proposed learning rule exemplified for the case of nonnegativity constraint.
• Figure 2: Weight kernels for networks trained with the proposed learning rule, the method of hopfield19, and BP, all incorporating the nonnegativity constraint.
• Figure 3: Comparison of synaptic weight distributions for networks trained with BP, the proposed learning rule, and hopfield19 under a nonnegativity constraint, fitted to lognormal, compressed, and stretched exponential distributions. Synaptic weights were normalized between 0 and 1 and clipped at 0.005 to account for the synaptic detection threshold. Tests showed that weights below this threshold had no significant impact on performance, while higher thresholds (e.g., 0.01) distorted the distributions. The chosen threshold of 0.005 preserved relevant synaptic connections without distorting the distribution.
• Figure 4: Comparison of test accuracy for networks trained with BP, the proposed learning rule, and hopfield19 under different conditions: Few-Shot Learning, Fast Gradient Attack, and Projected Gradient Descent. The x-axis represents the number of shots (left) and perturbation magnitude $\epsilon$ (middle, right). For the PGD attack, we used a step size of 0.01, a total of 40 iterations, and an $\ell_{\infty}$ norm constraint.