Self-orthogonalizing attractor neural networks emerging from the free energy principle

TL;DR

This work derives self-organizing attractor neural networks directly from the Free Energy Principle by employing deep particular partitions, showing how local free energy minimization yields both inference and learning rules that produce approximately orthogonal attractor representations. The framework unifies Boltzmann-machine–like stationary dynamics with non-equilibrium, sequence-capable flows through symmetric and antisymmetric couplings, enabling macro-scale Bayesian inference via sampling. Empirically, simulations demonstrate orthogonal basis formation, robust generalization, sequence learning, and resistance to catastrophic forgetting, underscoring potential benefits for brain-inspired AI and neuromorphic architectures. Overall, the paper provides a principled, hierarchical account of how adaptive, self-orthogonalizing attractor networks can emerge from first principles and solve core computational challenges in perception, memory, and learning.

Abstract

Attractor dynamics are a hallmark of many complex systems, including the brain. Understanding how such self-organizing dynamics emerge from first principles is crucial for advancing our understanding of neuronal computations and the design of artificial intelligence systems. Here we formalize how attractor networks emerge from the free energy principle applied to a universal partitioning of random dynamical systems. Our approach obviates the need for explicitly imposed learning and inference rules and identifies emergent, but efficient and biologically plausible inference and learning dynamics for such self-organizing systems. These result in a collective, multi-level Bayesian active inference process. Attractors on the free energy landscape encode prior beliefs; inference integrates sensory data into posterior beliefs; and learning fine-tunes couplings to minimize long-term surprise. Analytically and via simulations, we establish that the proposed networks favor approximately orthogonalized attractor representations, a consequence of simultaneously optimizing predictive accuracy and model complexity. These attractors efficiently span the input subspace, enhancing generalization and the mutual information between hidden causes and observable effects. Furthermore, while random data presentation leads to symmetric and sparse couplings, sequential data fosters asymmetric couplings and non-equilibrium steady-state dynamics, offering a natural extension to conventional Boltzmann Machines. Our findings offer a unifying theory of self-organizing attractor networks, providing novel insights for AI and neuroscience.

Figures (7)

• Figure 1: Deep Particular Partitions. A Schematic illustration of a particular partition of a system into internal ($\mu$) and external states ($\eta$), separated by a Markov blanket consisting of sensory states ($s$) and active states ($a$). The tuple $(\mu, s, a)$ is called a particle. A particle, in order to persist for an extended period of time, will necessarily have to maintain its Markov blanket, a behavior that is equivalent to an inference process in which internal states infer external states through the blanket states. The resulting self-organization of internal states corresponds to perception, while actions link the internal states back to the external states. B The internal states $\mu \subset x$ can be arbitrarily complex. Without loss of generality, we can consider that the macro-scale $\mu$ can be decomposed into set of overlapping micro-scale subparticles ($\sigma_i, s_i, a_i, s_{ij}, a_{ij}$), so that the internal state of subparticle $\sigma_i \subset \mu$ can be an external state from the perspective of another subparticle $\sigma_j \subset \mu$. Some, or all subparticles can be connected to the macro-scale external state $\eta$, through the macro-scale Markov blanket, giving a decomposition of the original boundary states into $s_i \subset s$ and $a_i \subset a$. The subparticles are connected to each other by the micro-scale boundary states $s_{ij}$ and $a_{ij}$. Note that this notation considers the point-of-view of the $i$-th subparticle. Taking the perspective of the $j$-th subparticle, we can see that $s_{ji}=a_{ij}$ and $a_{ji}=s_{ij}$. While the figure depicts the simplest case of two nested partitions, the same scheme can be applied recursively to any number of (possibly nested) subparticles and any coupling structure amongst them.
• Figure 2: Parametrization of subparticles in a deep particular partition. The internal state $\sigma_i$ of subparticle $\pi_i$ follows a continuous Bernoulli distribution, (a.k.a. a truncated exponential distribution supported on the interval $[ -1, +1] \subset \mathbb{R}$, see Appendix 1), with a prior "bias" $b_i$ that can be interpreted as a priori log-odds evidence for an event (stemming from a macro-scale sensory input $s_{i}$ - not shown, or from the internal dynamics of $\sigma_i$ itself, e.g. internal sequence dynamics). The state $\sigma_i$ is coupled to the internal state of another subparticle $\sigma_j$ through the micro-scale boundary states $s_{ij}$ and $a_{ij}$. The boundary states simply apply a deterministic scaling to their respective $\sigma$ state, with a weight ($J_{ij}$) implemented by a Dirac delta function shifted by $J_{ij}$ (i.e. we deal with conservative subparticles, in the sense of Friston_2023). The state $\sigma_i$ is influenced by its sensory input $s_i$ in a way that $s_i$ gets integrated into its internal bias, updating the level of evidence for the represented event.
• Figure 3: Free energy minimizing, adaptively self-organizing attractor network A Schematic of the network illustrating inference and learning processes. Inference and learning are two faces of the same process: minimizing local variational free energy (vfe), leading to dissipative dynamics and approximately orthogonal attractors. B A demonstrative simulated example (https://pni-lab.github.io/fep-attractor-network/simulation-demo) of the network's attractors forming an orthogonal basis of the input data. Training can be performed by introducing the training data (top left) through the biases of the network. In this example, the input data consists of two correlated patterns (Pearson's r = 0.77). During repeated updates, micro-scale (local, node-level) vfe minimization implements a simultaneous learning and inference process, which leads to approximate macro-scale (network-level) free energy minimization (bottom graph). The resulting network does not simply store the input data as attractors, but it stores approximately orthogonalized varieties of it (top right, Pearson's r = -0.19). C When the trained network is introduced a noisy version of one of the training patterns (left), it is internally handled as the Likelihood function, and the network performs an Markov-Chain Monte-Carlo (MCMC) sampling of the posterior distribution, given the priors defined by the network's attractors (top right), which can be understood as a retrieval process. D Thanks to its orthogonal attractor representation, the network is able to generalize to new patterns - as long as they are sampled from the sub-space spanned by the attractors - by combining the quasi-orthogonal attractor states (bottom right) by multistable stochastic dynamics during the MCMC sampling.
• Figure 4: Adaptive self-organization and generalization in a free-energy minimizing attractor network. Simulation results from training the network on a single, handwritten example for each of the 10 digits (0-9), with variations in training precision and evidence strength to explore different learning regimes (https://pni-lab.github.io/fep-attractor-network/simulation-digits). A: Performance landscapes as a function of inference temperature (inverse precision) and training evidence strength (bias magnitude). Retrieval performance (reconstructing noisy variants of the 10 training patterns, top left), one-shot generalization (reconstructing a noisy variants of unseen handwritten digits, top right), attractor orthogonality (mean squared angular difference from 90° indicating higher orthogonality for lower values, bottom left), and the number of attractors (when initialized with the 10 training patterns, bottom right) are shown. Optimal regions (contoured) highlight parameter settings that yield good generalization and highly orthogonal attractors. Contours in the top left and top right highlight the most efficient parameter settings for retrieval and generalization, respectively. Both contours are overlaid on the two bottom plots. B: Conceptual illustration of training regimes. With low temperature (high precision) high model complexity is allowed ("accuracy pumping") and attractors will tend to exactly match the training data. On the contrary, high temperatures (low precision) result in a single fixed point attractor and reduced recognition performance. However, such networks will be able to generalize to new data, suggesting the existance of "soft attractors" (e.g. saddle-like structures) that are not local minima on the free energy landscape, yet affect the steady-state posterior distribution in a non-negligible way (especially with longer mixing-times).
• Figure 5: (continued) A balanced regime can be found with intermediate precision during training, where both recognition and generalization performance are high. This is exactly the regime that promotes attractor orthogonalization, crucial for efficient representation and generalization. The complexity restrictions on these models cause them to re-use the same attractors to represent different patterns (see e.g. the single attractor belonging to the digits 5 and 7 in the example on panel D), which eventually leads to approximate orthogonality. Panels C-E provide examples of network behavior on a handwritten digit task across different regimes, including (i) training data (same in all cases); (ii) fixed-point attractors (obtained with deterministic update); (iii) attractor-orthogonality (polar histogram of the pairwise angles between attractors); (iv) retrieval and 1-shot generalization performance ($R^2$ between the noisy input pattern and the network output after 100 time steps, for 100 randomly sampled patterns) and (v) illustrative example cases from the recognition and 1-shot generalization tests (noisy input, network output and true pattern). C: High complexity: Attractors are sharp and similar to training data; good recognition, limited generalization. D: Balanced complexity (orthogonalization): Attractors are distinct and quasi-orthogonal, enabling strong recognition and generalization from noisy inputs. The balanced regime clearly demonstrates the network's ability to form an orthogonal basis, facilitating effective generalization as predicted by the free-energy minimization framework. E: Low complexity: There is only a single fixed-point attractor. Recognition performance is lower, but generalization remains considerable.