Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Nature-Inspired Local Propagation

Alessandro Betti, Marco Gori

TL;DR

It is shown that the algorithmic interpretation of the derived "laws of learning", which takes the structure of Hamiltonian equations, reduces to Backpropagation when the speed of propagation goes to infinity, which opens the doors to machine learning studies based on full on-line information processing that are based the replacement of Backpropagation with the proposed spatiotemporal local algorithm.

Abstract

The spectacular results achieved in machine learning, including the recent advances in generative AI, rely on large data collections. On the opposite, intelligent processes in nature arises without the need for such collections, but simply by online processing of the environmental information. In particular, natural learning processes rely on mechanisms where data representation and learning are intertwined in such a way to respect spatiotemporal locality. This paper shows that such a feature arises from a pre-algorithmic view of learning that is inspired by related studies in Theoretical Physics. We show that the algorithmic interpretation of the derived "laws of learning", which takes the structure of Hamiltonian equations, reduces to Backpropagation when the speed of propagation goes to infinity. This opens the doors to machine learning studies based on full on-line information processing that are based the replacement of Backpropagation with the proposed spatiotemporal local algorithm.

Nature-Inspired Local Propagation

TL;DR

It is shown that the algorithmic interpretation of the derived "laws of learning", which takes the structure of Hamiltonian equations, reduces to Backpropagation when the speed of propagation goes to infinity, which opens the doors to machine learning studies based on full on-line information processing that are based the replacement of Backpropagation with the proposed spatiotemporal local algorithm.

Abstract

The spectacular results achieved in machine learning, including the recent advances in generative AI, rely on large data collections. On the opposite, intelligent processes in nature arises without the need for such collections, but simply by online processing of the environmental information. In particular, natural learning processes rely on mechanisms where data representation and learning are intertwined in such a way to respect spatiotemporal locality. This paper shows that such a feature arises from a pre-algorithmic view of learning that is inspired by related studies in Theoretical Physics. We show that the algorithmic interpretation of the derived "laws of learning", which takes the structure of Hamiltonian equations, reduces to Backpropagation when the speed of propagation goes to infinity. This opens the doors to machine learning studies based on full on-line information processing that are based the replacement of Backpropagation with the proposed spatiotemporal local algorithm.
Paper Structure (21 sections, 6 theorems, 48 equations, 4 figures)

This paper contains 21 sections, 6 theorems, 48 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Recurrent Neural Networks and spatiotemporal locality
  3. Temporal locality and causality
  4. Spatial locality
  5. Learning as a Variational Problem
  6. Optimal Control Approach
  7. Recovering spatio-temporal locality
  8. Temporal Locality
  9. Spatial Locality
  10. From boundary to Cauchy's conditions
  11. Time Reversal of the Costate
  12. Continuous Time Reversal of State and Costate
  13. Hamiltonian Track
  14. Hamiltonian Sign Flip Strategy and time reversal
  15. Conclusions
  16. ...and 6 more sections

Key Result

Theorem 1

Let $H$ be as in equation eq:hamiltonian-general and assume that $x(0)=x^0$ and ${\boldsymbol{w}}(0)={\boldsymbol{w}}^0$ are given. Then a minimum of the functional in equation eq:funtional-G satisfies the Hamilton equations: together with the boundary conditions

Figures (4)

  • Figure 1: Recurrent net with $5$ neurons, $q=100$ (accuracy term), $r_w=1$ (weight regularization term), $r= 0.1$ (derivative of the weight term).
  • Figure 2: Recurrent net with $5$ neurons, $q=1000$ (accuracy term), $r_w=1$ (weight regularization term), $r= 0.1$ (derivative of the weight term)..
  • Figure 3: Tracking a highly-predictable signal: number of neurons: $5$, $q = 100$ (accuracy), weight reg = $1$, derivative of weight reg = $0.1$
  • Figure 4: Evolution of the Lagrangian and of the Hamiltonian function for the experiment whose tracking is shown in the left-side figure.

Theorems & Definitions (20)

  • proof
  • Theorem 1: Hamilton equations
  • proof
  • Theorem 2
  • proof
  • Remark 1
  • proof
  • Corollary 1: Reduction to Backprop
  • proof
  • Example 1
  • ...and 10 more