Knowledge as a Breaking of Ergodicity

TL;DR

The work reframes knowledge acquisition from binary data as a thermodynamic problem, introducing a high-order Ising-like energy $E(oldsymbol{\sigma})$ whose couplings $\mathbf{J}$ are learned from dataset weights via $J = -2^{-N}\sum_i \boldsymbol{\sigma}_i E_i$, and retrieval proceeds with Gibbs sampling at temperature $T$. It constructs a conjoint free-energy surface $A({E_i},T)$ and a Legendre-transformed version over coarse-grained weights $x_i$, linking learning and retrieval to minimization on a thermodynamic landscape and highlighting a Gibbs-inequality bound. The central finding is that reducing description to a smaller set of couplings induces multiple free-energy minima, fracturing the configuration space into ergodic subspaces and creating kinetic bottlenecks that complicate learning and retrieval; this ergodicity breaking is analogous to phase coexistence and requires remedies such as parameterizing non-represented (unseen) states with an extensive energy gap and possibly deploying multiple expert models for distinct minima. These insights connect to broader themes in physics-inspired inference, inform strategies for robust, context-aware knowledge libraries, and suggest practical parallels to force-field design and protein-folding-like landscape funneling in complex systems.

Abstract

We construct a thermodynamic potential that can guide training of a generative model defined on a set of binary degrees of freedom. We argue that upon reduction in description, so as to make the generative model computationally-manageable, the potential develops multiple minima. This is mirrored by the emergence of multiple minima in the free energy proper of the generative model itself. The variety of training samples that employ N binary degrees of freedom is ordinarily much lower than the size 2^N of the full phase space. The non-represented configurations, we argue, should be thought of as comprising a high-temperature phase separated by an extensive energy gap from the configurations composing the training set. Thus, training amounts to sampling a free energy surface in the form of a library of distinct bound states, each of which breaks ergodicity. The ergodicity breaking prevents escape into the near continuum of states comprising the high-temperature phase; thus it is necessary for proper functionality. It may however have the side effect of limiting access to patterns that were underrepresented in the training set. At the same time, the ergodicity breaking within the library complicates both learning and retrieval. As a remedy, one may concurrently employ multiple generative models -- up to one model per free energy minimum.

Figures (12)

• Figure 1: The free energy surface $\widetilde{A}(J_1, J_2, J_{12})$ from Eq. (\ref{['Atilde']}), as a function of the coupling constants, for the standard model (\ref{['ccE2']}); $M=2^N=4$. Two select cross-sections $J_{12} = \text{const}$ are shown. The $J_{12} = -J$ cross-section contains the global minimum of $\widetilde{A}$, while the $J_{12} = 0$ cross-section corresponds to a reduced description in which the two-body interaction is missing. $T = T^\circ=1$, $J=1.5$.
• Figure 2: Panels (a) and (c) provide contour plots corresponding to the $J_{12} = 0$ and $J_{12} = - J = -1.5$ slices, respectively, from Fig. \ref{['AE2fig']}. Panel (b) exemplifies a special intermediate situation, where the surface $\widetilde{A}(J_1, J_2, J_{12})$ just begins to develop two distinct minima. $T = T^\circ=1$, $J=1.5$.
• Figure 3: Dashed lines: the $m \equiv m_1 = - m_2$ slice of the free energy surface $\widetilde{A}(m_1, m_2)$ from Eq. (\ref{['FMF3']}) for three select values of $J$. Solid lines: the respective values of the exact Helmholtz energy $A$ for the energy function (\ref{['E2']}). $T=T^\circ=1$.
• Figure 4: The two branches of the restricted Gibbs energy $\widetilde{G}$ are labeled with $\widetilde{G}^-$ and $\widetilde{G}^+$. The equilibrium Gibbs energy $\widetilde{G}_\text{eq}$ from Eq. (\ref{['Gcg']}) is shown for select values of $N_r$. The $N_r=64$ curve is masked by the meanfield curves. $J=1.5$, $T=T^\circ=1$.
• Figure 5: The magnetization, per spin, as a function of the source field along the slice $m=m_1=-m_2$, $h=h_1=-h_2$, below the critical point. The meanfield curves correspond to the $m_1 > 0$, $m_2 < 0$ minimum (red), the $m_1 < 0$, $m_2 > 0$ minimum (blue), and the mechanically unstable region separating the spinodals (pink); the portions of this curve where $mh <0$ are either metastable or unstable. The equilibrium values for the energy function (\ref{['Exz']}) are given for several system sizes. $T=T^\circ=1$, $J=1.5$.