Dense Associative Memory with biased patterns: a Replica Symmetric analysis

Abstract

We investigate dense higher-order associative memories in the high storage regime when the stored patterns are biased, namely when the entries of the patterns are not symmetrically distributed around zero. In this setting, the standard Hebbian prescription must be modified by recentering and rescaling the pattern entries, and an additional term must be introduced in the Hamiltonian to enforce consistency between the average activity of the network and that of the stored patterns. As a first step, we perform a signal-to-noise analysis in the zero-temperature limit and show that the bias reduces the effective storage capacity through a multiplicative correction factor (1-b^2)^P, while preserving the superlinear scaling with the system size. We then derive the quenched statistical pressure within the Replica Symmetric framework by means of Guerra's interpolation method and obtain the corresponding self consistency equations for the relevant order parameters. The analytical treatment confirms the heuristic prediction of the signal-to-noise argument, showing that the same bias dependent renormalization naturally emerges in the variance of the cross-talk noise. Finally, we discuss the resulting phase behavior of the model and its implications for retrieval performance in the model.

Figures (5)

• Figure 1: Representation of a toy example of the network under investigation with $P=3$ grade of interactions, $N=7$ spins. Vertices correspond to spins $\sigma_i$, and each triangle encodes a term $\sigma_{i_1}\sigma_{i_2}\sigma_{i_3}$ weighted by the biased patterns $\bm \eta^\mu=\bm \xi^\mu - b$. Highlighted triangles represent selected contributions, while gray ones denote the remaining possible interactions. The global term proportional to $(\sum_{i=1}^N \sigma_i - Nb)^2$ is also indicated.
• Figure 2: Numerical resolution of zero temperature self consistency equation is presented. The upper row corresponds to $P=4$, and the lower row to $P=6$. In the first column, $\gamma=0.1$ is held fixed, while different values of $b$ and $g$ are explored. In the second column, $g=1.0$ is fixed, and both the load $\gamma$ and bias $b$ are varied.
• Figure 3: Behaviour of the critical load of the network at null temperature $T =0$ for different values of $g$ and $b$. In each panel we change the grade of interactions of the network (from left to right, $P=4,6,8$) and we analyze different values of $g$ from $g=0$, when the term tuning the constraint on the activity of the network vanishes, to $g=1$. We can see that $g$ plays the role of a light booster for the critical load, whose contribution decreases when the degree of interactions increases. Moreover, the critical load becomes smaller as the bias of the patterns increases.
• Figure 4: MCMC simulation, for $N=1000$, $K=O(N^2)$, $b=0.3$ and $g=1.0$, of the Mattis magnetization $m(\sigma)$ and the mean activity $M(\sigma)$ with respect to different values of the degree of interactions $P$. Starting from a configuration which has an overlap with $\xi^1$ equal to $0.6$, the standard Hopfield case $(P=2)$ is not able to retrieve the pattern, whereas the dense associative memory with $P=4$ does this and, moreover, the mean activity stabilizes around the value of the bias $b$, as expected.
• Figure 5: MCMC simulation for $N=1000$, $K= 0.05 N^3$, $b=0.3$, $g=1.0$. In the dense, low-load limit, the system successfully retrieves the stored pattern even in the presence of bias, yielding both $m(\sigma)=1$ and $M(\sigma)=b$. Conversely, for $P=2$ retrieval fails, as the load exceeds the network storage capacity. In all cases, the dynamics are initialized from a network configuration such that $\bm{\sigma}^{(0)} \cdot \bm{\xi}^1 \approx 1.0$. For $P=2$, the trajectory rapidly departs from the initial condition, whereas for $P=4$ the system remains trapped in the initial configuration, indicating stability.