Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neural Learning Rules from Associative Networks Theory

Daniele Lotito

TL;DR

This work places neural learning rules within a rigorous energy-based, associative-network framework that leverages a multi-time-scale architecture and a generating function $\mathcal{L}$. By allowing $\mathcal{L}$ to depend on a memory matrix $\Xi$, it recovers conventional Hebbian couplings via $J = \Xi^{\top} \Xi / N_h$ and introduces a synaptic-memory dynamics that enables learning and memory updating. The authors connect their framework to Hopfield network theory, derive conditions for hyperbolic fixed points through random-matrix analyses (e.g., Wishart spectra and Marchenko-Pastur bounds), and show that under appropriate limits the model reproduces classic Hebbian learning while supporting broader, biologically plausible synaptic dynamics. Overall, the paper provides a solid mathematical foundation and broadens the applicability of associative-network theory to learning in neural systems, with potential cross-disciplinary implications in dynamical systems and probabilistic modeling.

Abstract

Associative networks theory is increasingly providing tools to interpret update rules of artificial neural networks. At the same time, deriving neural learning rules from a solid theory remains a fundamental challenge. We make some steps in this direction by considering general energy-based associative networks of continuous neurons and synapses that evolve in multiple time scales. We use the separation of these timescales to recover a limit in which the activation of the neurons, the energy of the system and the neural dynamics can all be recovered from a generating function. By allowing the generating function to depend on memories, we recover the conventional Hebbian modeling choice for the interaction strength between neurons. Finally, we propose and discuss a dynamics of memories that enables us to include learning in this framework.

Neural Learning Rules from Associative Networks Theory

TL;DR

This work places neural learning rules within a rigorous energy-based, associative-network framework that leverages a multi-time-scale architecture and a generating function . By allowing to depend on a memory matrix , it recovers conventional Hebbian couplings via and introduces a synaptic-memory dynamics that enables learning and memory updating. The authors connect their framework to Hopfield network theory, derive conditions for hyperbolic fixed points through random-matrix analyses (e.g., Wishart spectra and Marchenko-Pastur bounds), and show that under appropriate limits the model reproduces classic Hebbian learning while supporting broader, biologically plausible synaptic dynamics. Overall, the paper provides a solid mathematical foundation and broadens the applicability of associative-network theory to learning in neural systems, with potential cross-disciplinary implications in dynamical systems and probabilistic modeling.

Abstract

Associative networks theory is increasingly providing tools to interpret update rules of artificial neural networks. At the same time, deriving neural learning rules from a solid theory remains a fundamental challenge. We make some steps in this direction by considering general energy-based associative networks of continuous neurons and synapses that evolve in multiple time scales. We use the separation of these timescales to recover a limit in which the activation of the neurons, the energy of the system and the neural dynamics can all be recovered from a generating function. By allowing the generating function to depend on memories, we recover the conventional Hebbian modeling choice for the interaction strength between neurons. Finally, we propose and discuss a dynamics of memories that enables us to include learning in this framework.

Paper Structure

This paper contains 20 sections, 46 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Main Results
  3. Structure of the paper
  4. Modeling assumptions
  5. Energy-based modeling
  6. The hidden neurons algebraic equation
  7. The coupling matrix and the memory matrix
  8. Multiple time scale modeling
  9. Discussion
  10. Theoretical Takeaways
  11. Generating functions that depend on the memory matrix
  12. Synaptic dynamics and Hebbian theory
  13. Connection to Hopfield model theory
  14. Recovering Hebbian learning with synaptic evolution
  15. Eigenvalue problems in multi scale dynamics
  16. ...and 5 more sections

Figures (1)

  • Figure 1: Evolution of cosine similarity between each memory row $\Xi_\mu(t)$ and its target pattern $\bm \xi^\prime_\mu$ as described by Eq.\ref{['eq:cosine']}. Solid lines represent numerical simulations obtained by simulating the synaptic dynamics (see Eq.\ref{['eq:memdyn_el']}) with parameters $N_v=50$, $N_h=4$, $\tau_\Xi=250$, $\beta=2$, and noise level $\sigma=0.4$. Initial memories are constructed to be perfectly orthogonal through a Gram-Schmidt process and scaled to norm $\sqrt{N_v}$. Dashed lines show the theoretical prediction from Eq.\ref{['eq:sim_final']}, where the characteristic time scales $\tau_\mu$ follow from Eq.\ref{['eq:tau']} with uniform probabilities $p_\mu=1/N_h$. The perfect agreement confirms our analytical understanding of the memory evolution process for a wide range of parameter values.