Thermodynamics of bidirectional associative memories

TL;DR

This work analyzes the equilibrium behavior of Bidirectional Associative Memories (BAMs), a bipartite generalization of Hopfield networks. It develops a statistical-mechanics framework based on replica theory and Guerra interpolation to derive replica-symmetric and one-step replica-symmetry-breaking phase diagrams in finite temperature and high-load regimes, including an analytic P-SG line and zero-temperature capacity. It shows that layer asymmetry reduces retrieval regions but can improve efficiency in terms of the number of weights stored, and it connects BAM to two coupled RBMs and to a low-load analogy with coupled Hopfield models. Numerical simulations validate the analytic predictions and reveal basins of attraction and retrieval mechanisms, highlighting BAM as a scalable, energy-based framework for generative-like learning.

Abstract

In this paper we investigate the equilibrium properties of bidirectional associative memories (BAMs). Introduced by Kosko in 1988 as a generalization of the Hopfield model to a bipartite structure, the simplest architecture is defined by two layers of neurons, with synaptic connections only between units of different layers: even without internal connections within each layer, information storage and retrieval are still possible through the reverberation of neural activities passing from one layer to another. We characterize the computational capabilities of a stochastic extension of this model in the thermodynamic limit, by applying rigorous techniques from statistical physics. A detailed picture of the phase diagram at the replica symmetric level is provided, both at finite temperature and in the noiseless regimes. Also for the latter, the critical load is further investigated up to one step of replica symmetry breaking. An analytical and numerical inspection of the transition curves (namely critical lines splitting the various modes of operation of the machine) is carried out as the control parameters - noise, load and asymmetry between the two layer sizes - are tuned. In particular, with a finite asymmetry between the two layers, it is shown how the BAM can store information more efficiently than the Hopfield model by requiring less parameters to encode a fixed number of patterns. Comparisons are made with numerical simulations of neural dynamics. Finally, a low-load analysis is carried out to explain the retrieval mechanism in the BAM by analogy with two interacting Hopfield models. A potential equivalence with two coupled Restricted Boltmzann Machines is also discussed.

Figures (11)

• Figure 1: Schematic representation of a BAM with $N=6$ and ${\bar{N}}=4$. The (binary) neurons are represented by black circles; each solid line defines a synaptic weight connecting two neurons of different layers.
• Figure 2: Phase diagram of the BAM in the plane $\left(\alpha,T\right)$ at different values of $\gamma$. (a): $\gamma=1$. (b): $\gamma=2$; (c): $\gamma=5$. The labels $\textbf{P}$, $\textbf{SG}$, $\textbf{R}$, $\textbf{MR}$ (shown only in (a)) stand for paramagnetic, spin-glass, retrieval and metastable-retrieval, respectively.
• Figure 3: Critical lines separating the different operating regimes of the BAM. In (a), critical lines for the MR-SG. In (b), critical lines for the R-MR transition at different values of $\gamma$. In (c), critical lines for the P-SG transition at different values of $\gamma$. All the lines are plotted as function of $\alpha$ (please mind the three panels cover different ranges of $\alpha$), for different values of $\gamma$. The inset in panel (c) shows a comparison of the P-SG critical lines between the symmetric BAM, given by \ref{['eq:T_SG-P_line_symmetric']}, and the same critical line computed for the Hopfield model (after a proper rescaling to take into account the different definition of the load in the two models).
• Figure 4: Panel (a): BAM's phase diagram at $T=0$ in the plane $\left(\alpha,\gamma\right)$: the same color-code as in Figure \ref{['fig:Phasediagrams_all']} is used. The phase boundary separating the MR (yellow) from the SG (violet) phases defines the RS critical capacity $\alpha_{c}$ as a function of $\gamma$. As the vertical axis has a logarithmic scale, we notice again how the phase diagram is symmetric under the transformation $\gamma\to\gamma^{-1}=\bar{\gamma}$. Panels (b) and (c) show the zero-temperature Mattis magnetizations of the two layers (plotted resp. in panels (b) and (c)) as functions of $\alpha$ for $4$ different values of $\gamma$.
• Figure 5: (a) Critical capacity of BAM and the Hopfield but normalized by the total number of neurons in the network (i.e. $N+\bar{N}$ instead of $L$ as in \ref{['eq:alpha']} for the BAM). The green line corresponds to Eq. \ref{['eq:criticalcapBAM_renormalized']}. (b) Number of weights used to store this critical number of patterns in the BAM normalized by the analogous number of weights in the Hopfield model. In both panels, each quantity is plotted as a function of the asymmetry $\gamma = \sqrt{N\slash \bar{N}}$.