Dependence of Equilibrium Propagation Training Success on Network Architecture

TL;DR

The paper tackles the energy and scalability challenges of AI by evaluating equilibrium propagation (EP) on physically plausible, locally constrained lattice architectures using an XY model. It combines XOR, Iris, and MNIST benchmarks to analyze how architecture influences EP training, illustrating that sparse lattices with local interconnections can match dense networks in several tasks, especially when skip connections enable long-range information transport. The work introduces a detailed view of how network responses and couplings evolve, revealing self-organization into an affected region and a marginal region and demonstrating that layered lattices (LCL and CNN-like) with local inter-layer connections can achieve competitive MNIST performance under realistic hardware constraints. These findings provide architecture-aware design guidelines for scaling EP-based neuromorphic systems and highlight the importance of skip connections and local inter-layer coupling in preserving computational capability.

Abstract

The rapid rise of artificial intelligence has led to an unsustainable growth in energy consumption. This has motivated progress in neuromorphic computing and physics-based training of learning machines as alternatives to digital neural networks. Many theoretical studies focus on simple architectures like all-to-all or densely connected layered networks. However, these may be challenging to realize experimentally, e.g. due to connectivity constraints. In this work, we investigate the performance of the widespread physics-based training method of equilibrium propagation for more realistic architectural choices, specifically, locally connected lattices. We train an XY model and explore the influence of architecture on various benchmark tasks, tracking the evolution of spatially distributed responses and couplings during training. Our results show that sparse networks with only local connections can achieve performance comparable to dense networks. Our findings provide guidelines for further scaling up architectures based on equilibrium propagation in realistic settings.

Figures (5)

• Figure 1: Lattice architectures and training performance for XOR: (a) Schematic descriptions of lattice architectures studied in this work and encoding of XOR. (b) Input-output node configuration. (c) Left: performance of lattices in training with EP, measured by the average distance function. Right: Distribution of distance at the last 100 epochs. We denote averages over input–output pairs by $\langle * \rangle$ and averages over different training cases by $\overline{*}$.
• Figure 2: Evolution of lattice response to flip on input nodes: (a) Schematic demonstration of the experiment. Nodes "In1" and "In2" are the two input nodes on the lattice. The value of In2 is flipped and the response of spins at other nodes are recorded. (b) Sample result of the experiment for the lattice response of the SQ and 4NSQ lattices. The two panels on the left show the lattice response to the flip on node In2 for the SQ and 4NSQ lattices before the training. The two panels on the right show the response after training. The panel at the center shows the evolution of the mean error over all the 4 input-output pairs for SQ (blue line) and 4NSQ (orange line), in a typical example. (c) The evolution of lattice response for different lattices at specific stage of training. (d) The panel on the left shows the evolution of response at the output node. The panel on the right shows the sum of response at all the nodes in the lattice, as an indicator of total intensity of response.
• Figure 3: Evolution of lattice response and couplings: (a) Comparison between typical training cases. On the left the connections in the SQ and 4NSQ lattices are shown. On the right the connections after training are shown. A region in which a significant change is observed is encircled. (b) The couplings of lattices of different architectures at different stages of training.
• Figure 4: Effects of network depth: (a) Schematic graph of the configuration of the tested lattices, taking 4NSQ as an example. (b) The interpretation of output configurations and their projection onto the triangular graphs. (c) The result of classification of training samples at different stages of training for lattices of different architecture and size. (d) Comparison between performance of lattices (solid lines) to all-to-all networks of the same number of parameters (dashed lines of the same colors) and densely coupled layered network (black dashed line).
• Figure 5: Architecture and performance of layered networks trained on MNIST, all with a single hidden layer. (a) Schematic of a locally coupled lattice (LCL) layer. (b) Schematic of LCL, CNN-like, and dense (DL) networks. (c) LCL network performance – top: effect of window size; middle: effect of strides; bottom: effect of intra-layer couplings. (d) Comparison of architectures: left, CNN-like networks with multiple channels versus LCL network with $6\times6$ window; right, LCL (solid lines) versus DL (dashed lines), with same-color lines indicating similar parameter counts. (e) Effect of parameter count on test accuracy. For LCL and LCL+IS models, labels of the form $m \times n$ indicate the window size, with the term following "+" denoting the intra-layer architecture. In CNN-like models, the filter size is fixed at $6 \times 6$, and $m$c indicates $m$ channels. In DL models, labels such as "1h21" represent one hidden layer with 21 nodes. The black dashed line indicates 92.3% test accuracy attained by a linear classifier.