Explosive neural networks via higher-order interactions in curved statistical manifolds

TL;DR

It is shown that these curved neural networks implement a self-regulating annealing process that can accelerate memory retrieval, leading to explosive order-disorder phase transitions with multi-stability and hysteresis effects.

Abstract

Higher-order interactions underlie complex phenomena in systems such as biological and artificial neural networks, but their study is challenging due to the scarcity of tractable models. By leveraging a generalisation of the maximum entropy principle, we introduce curved neural networks as a class of models with a limited number of parameters that are particularly well-suited for studying higher-order phenomena. Through exact mean-field descriptions, we show that these curved neural networks implement a self-regulating annealing process that can accelerate memory retrieval, leading to explosive order-disorder phase transitions with multi-stability and hysteresis effects. Moreover, by analytically exploring their memory-retrieval capacity using the replica trick, we demonstrate that these networks can enhance memory capacity and robustness of retrieval over classical associative-memory networks. Overall, the proposed framework provides parsimonious models amenable to analytical study, revealing higher-order phenomena in complex networks.

Figures (7)

• Figure 1: Higher-order decomposition resulting from the foliation of a statistical manifold. Illustration of a family of standard MEP models (right) and its deformed counterpart (bottom left). The space of MEP distributions with constraints of different orders constitute nested sub-manifolds amari2001information, giving rise to a hierarchy of sub-families of models of the form $\mathcal{E}_k^\gamma = \{ p_\gamma^{(k)}(\bm x) = e^{-\varphi_{\gamma}} [1 - \gamma \beta E_k(\bm x) ]_+^{1/\gamma} \}$ such that $\mathcal{E}_1^\gamma\subset\mathcal{E}_2^\gamma\subset\dots\subset\mathcal{E}_n^\gamma$morales2021generalization. The foliation depends on the curvature $\gamma$, and in general $\mathcal{E}_k^\gamma \neq \mathcal{E}_k^0$ but rather $\mathcal{E}_k^\gamma \cap \mathcal{E}_r^0 \neq \varnothing$ for $k < r$. For small values of $|\gamma|$, it is possible to neglect higher-order terms in \ref{['eq:deformed_exp_expansion']}, and therefore certain subsets of $\mathcal{E}_k^\gamma$ effectively approximate $\mathcal{E}_r^0$.
• Figure 2: Explosive phase transitions in curved neural networks. (a) Phase transitions of the curved neural network with one associative memory, for $J=1$ and values of $\gamma'=-0.5$ (top, displaying a second-order phase transition) and $\gamma'=-1.5$ (bottom, displaying an explosive phase transition). Solid lines represent the stable fixed points, and dotted lines correspond to unstable fixed points. (b) Phase diagram of the system. The areas indicated by P and M refer to the usual paramagnetic (disordered) and magnetic (ordered) phases, respectively. The area indicated by Exp represents a phase where ordered and disordered states coexist in an explosive phase transition characterised by a hysteresis loop. (c) Solutions of (\ref{['eq:mf_one-pattern']}-\ref{['eq:beta_one-pattern']}) for $\beta',m,\beta$ (black line) for $\gamma'=-1.2$, and projections to the plane $m=0$, $\beta=0$ and $\beta'=0$, obtaining respectively the relation between $\beta,\beta'$ and solutions of the flat and the deformed models respectively (grey lines). (d) Mean-field dynamics of the single-pattern neural network for $\beta=1.001$ (near criticality from the ordered phase) for some values of $\gamma'$ in $[-1.5,0]$. For large negative $\gamma'$ the dynamics 'explodes', with $m$ (top) and $\beta'$ (bottom) converging abruptly.
• Figure 3: Interaction between two encoded memories.(a) Values of $\varphi_{\gamma}$ for different mean-field values $m_1,m_2$, indicating the attractor structure of the network for different values of $\beta$ with $J=1, C=0.2$ for $\gamma'=0$ (top row) and $\gamma'=-1.2$ (bottom row). (b) Bifurcations of the order parameters $m_1,m_2$. For $\gamma'=0$ we observe an attractor bifurcating into two and then into four. For $\gamma'=-1.2$, we observe the same sequence, but with a coexistence hysteresis regime in which 7 attractors are possible.
• Figure 4: Memory capacity is enhanced by geometric deformation. Phase diagram of a curved associative memory with an extensive number of encoded patterns $M=\alpha N$ and $J=1$ for (a) different $T=1/\beta$ at $\gamma'=0$ (black dashed lines), $0.8, -0.8$ (solid lines), and for (b) different $\gamma'$ at $\beta=2$. F indicates the ferromagnetic (i.e., memory retrieval) phase, SG the spin-glass phase (where saturation makes memory retrieval inviable), M a mixed phase, and P the paramagnetic region. Both in F and M, ferromagnetic and spin-glass solutions coexist, but we differentiate these by calculating respectively whether memory-retrieval or spin-glass solutions are the global minimum of the normalising potential $\varphi_\gamma$. The dotted lines in (a) near $T=0$ indicate the AT lines, below which the replica-symmetric solution is not valid. Increasing $\gamma'$ to larger negative values extends the retrieval phase into larger values of $\alpha$, indicating an increased memory capacity, while larger positive values reduce the extension of the mixed phase, increasing robustness of memory retrieval.
• Figure 5: Simulation study for the effect of deformation on image encoding.(a) Examples of CIFAR-100 images (top) and their RGB binarised versions (bottom). Every 32x32x3 binary RGB pixel value for each image $a$ is assigned to the value of one position of pattern $\xi_i^a$. (b, c) Mean and variance of pattern retrieval values obtained in experiments, measured by the overlap between the final state of the network and the encoded pattern.