Serial vs parallel recall in the Blume-Every-Griffiths neural networks

TL;DR

In order to implement graded responses in the neurons, variations on theme obtained by enlarging the possible values of neural activity these neurons may sustain are discussed generalizing the Ghatak-Sherrington model for inverse freezing in Hebbian terms.

Abstract

Fully connected Blume-Emery-Griffiths neural networks performing pattern recognition and associative memory have been heuristically studied in the past (mainly via the replica trick and under the replica symmetric assumption) as generalization of the standard Hopfield reference. In these notes, at first, by relying upon Guerra interpolation, we re-obtain the existing picture rigorously. Next we show that, due to dilution in the patterns, these networks are able to switch from serial recall (where one pattern is retrieved per time) to parallel recall (where several patterns are retrieved at once) and the larger the dilution, the stronger this emerging multi-tasking capability. In particular, we inspect the regimes of mild dilution (where solely a low storage of pattern can be enabled) and extreme dilution (where a medium storage of patterns can be sustained) separately as they give rise to different outcomes: the former displays hierarchical recall (distributing the amplitudes of the retrieved signals with different amplitudes), the latter -- instead -- performs a equal-strength recall (where a O(1) fraction of all the patterns is simultaneously retrieved with the same amplitude per pattern). Finally, in order to implement graded responses in the neurons, variations on theme obtained by enlarging the possible values of neural activity these neurons may sustain are also discussed generalizing the Ghatak-Sherrington model for inverse freezing in Hebbian terms.

Figures (9)

• Figure 1: MCMC outcomes: Mattis magnetizations for $N=1000$ and $K=6$ diluted patterns, shown at $\beta=1000$ for three activity levels: $a=0.05$ (left), $a=0.1$ (center), and $a=0.6$ (right). Rows correspond to: (top) diluted Hopfield, (middle) BEG, and (bottom) GS with $S=1.5$. Bars report the steady-state overlaps $m_\mu$ measured in Monte Carlo simulations, while the orange dotted line shows the hierarchical prediction $m_\ell=\mathcal{N}_1(1-a)^{\ell-1}$ in \ref{['eq:hierarc_ansatz']}. The diluted Hopfield model closely follows the hierarchical profile for all $a$. When the additional $\eta$-sector is present (BEG and GS), deviations from the hierarchical ordering grow as dilution is reduced (larger $a$), and the number of non-vanishing overlaps drops more abruptly than in the diluted Hopfield case. This is explained by the energetic competition discussed in Sec. \ref{['subsec:eta_term']}.
• Figure 2: Energy comparison between the pure state \ref{['eq:pure_state']} and the hierarchical/parallel state \ref{['eq:parallel_state']}. Solid lines are the analytical predictions \ref{['eq:num_puro']}--\ref{['eq:num_ger']}, while dots are numerical simulations with $N=2000$ and $K=10$. Top row: $S=1.0$, $\bm{\sigma}\in\{-1,0,1\}^{N}$. Bottom row: $S=1.5$, $\bm{\sigma}\in\{-1,-1/3,1/3,1\}^{N}$. Columns show: (i) the $\xi$-dependent contribution $|\mathcal{H}^{(\xi)}|$, (ii) the $\eta$-dependent contribution $|\mathcal{H}^{(\eta)}|$, and (iii) the total energy. The $\xi$-term always favors the hierarchical/parallel state, while the $\eta$-term can favor the pure state at weak dilution when $0\in\Omega$ (integer $S$), producing a crossover in the total energy for $S=1$; by contrast, for $S=1.5$ (half-integer $S$) the parallel state remains energetically preferred across essentially the whole range of $a$.
• Figure 3: MCMC simulations of \ref{['eq:HamiltSspin']} with $K=5$, $\beta=1000$, $N=1000$, and $a=0.6$. The pre-factor $A$ multiplies the $\xi$-dependent term, while $B$ multiplies the $\eta$-dependent term. Top row:$0\in\Omega$. Increasing $B$ progressively suppresses multitasking until only one magnetization remains non-zero. Bottom row:$0\notin\Omega$. The same increase in $B$ has a much weaker impact, and multiple overlaps can coexist even at larger $B$.
• Figure 4: Schematic representation of the BEG neural network (left) and the corresponding generalized restricted Boltzmann machine (right). In the recurrent neural network on the left, for each pair of the neurons there are two contributions within their couplings: the former (highlighted in blue) accounts for the interactions with $\bm \xi$ while the latter accounts for those labeled by $\bm \eta$ (highlighted in red). Instead, in the dual RBM, each neuron $\sigma$ interacts with two different layers --of the same width $K$-- built off by Gaussian neurons $\bm z$ and $\bm \tilde{z}$: one layer conveys interactions mitigated by $\bm \xi$, the other accounts for those pertaining to the $\bm \eta$ as prescribed accordingly to eq. \ref{['eq:parition_start_BEG']}. Note, in both the networks, the presence of autapses on the $\bm\sigma$ layer, that is, self-couplings in statistical mechanical jargon GalliusGalliusTwo.
• Figure 5: MCMC simulations (dots) versus solutions of the self-consistency equations \ref{['eq:selfMT1']} (solid lines), for $K=2$ stored patterns and $N=3000$ spins. Top row: $S=1$ (i.e. $\boldsymbol\sigma\in\{-1,0,+1\}^N$). Bottom row: $S=1.5$ (i.e. $\boldsymbol\sigma\in\{-1,-\tfrac{1}{3},+\tfrac{1}{3},+1\}^N$). The agreement between simulations and theory is excellent across all regimes. The self-consistency equations correctly predict the emergence of a parallel retrieval ansatz in certain dilution regions, i.e. both Mattis magnetizations remain simultaneously non-zero. For small $\beta$ (high temperature), parallel retrieval is observed only at strong dilution (small $a$), regardless of whether $0\in\Omega$. For large $\beta$ (low temperature), parallel retrieval persists for all values of $a$ when $0\notin\Omega$, whereas if $0\in\Omega$ it survives only up to a critical dilution level. In particular, for $K=2$, $S=1$ and $\beta=500$, the critical threshold is $a_c\simeq 0.5$.

Theorems & Definitions (39)

• Definition 1: BEG Hamiltonian
• Remark 1: Normalization and basic statistics
• Definition 2: Partition function
• Definition 3: Gibbs measure
• Definition 4: Replicated system
• Definition 5: Global average
• Definition 6: Quenched statistical pressure / Quenched free energy
• Definition 7: Control parameters
• Definition 8: Order parameters
• Remark 2: On the non-dense character of the model