When low-loss paths make a binary neuron trainable: detecting algorithmic transitions with the connected ensemble

TL;DR

This work introduces the connected ensemble as a statistical-mechanics framework to locate low-loss, trainable regions in rugged landscapes and applies it to the symmetric binary perceptron (SBP). By constructing a chain of connected minima with fixed overlap $m$ and employing a coarse-graining approach to access the $m\to1$ saddle point beyond the no-memory Ansatz, the authors identify an easy-training regime bounded by $α_{\rm connected}$ and $κ_{\rm connected}$. They reveal an emergent, translation-invariant path correlation with an exponential decay characterized by a correlation length $\xi$ that diverges at the connected transition, and they show margin distributions that differentiate robust core and edge minima. The results offer a principled route to design local algorithms that exploit connected manifolds and raise questions about non-annealed effects and extensions to broader models in rugged optimization.

Abstract

We study the connected ensemble, a statistical-mechanics framework that characterizes the formation of low-loss paths in rugged landscapes. First introduced in a previous paper, this ensemble allows one to identify when a network can be trained on a simple task and which minima should be targeted during training. We apply this new framework to the symmetric binary perceptron model (SBP), and study how its typical {connected} minima behave. We show that {connected} minima exist only above a critical threshold $κ_{\rm connected}$, or equivalently below a critical constraint density $α_{\rm connected}$. This defines a parameter range in which training the network is easy, as local algorithms can efficiently access this connected manifold. We also highlight that these minima become increasingly robust and closer to one another as the task on which the network is trained becomes more difficult.

Figures (7)

• Figure 1: Illustration of the coarse-graining approach, focusing on the entropic partition function $\mathcal{Z}_{S}$ in the connected free energy. At the top, we have the generic field matrix $\hat{\bf Q}$ that couples all the binary elements $\{x^k\}_{k\in[\![1,k_f]\!]}$ -represented with the colored lines-. The first step of the coarse-graining is to consider that only a subgrid of elements $\{x^{*,k}\}_{k\in[\![1,k_f^*]\!]}$ can have generic interactions -represented with the remaining black and green lines-. The remaining binary elements follow a no-memory interactions profile -red lines- to ensure the connected structure. Because these elements only interacts with their nearest-neighbors, they can be integrated analytically. This second step generates a new effective potential $\phi_S$ between neighboring elements ${\bf x}^{*,k}/{\bf x}^{*,{k\pm 1}}$. The detailed computation for this coarse-graining approach can be found in App. \ref{['app: coarse-graining']}.
• Figure 2: Evolution of the correlation profile ${\bf Q}^*_{k,k'}$ in the core of the connected manifold (i.e., $k'=k_f^*/2$). In both plots we increase $\alpha$ while keeping $\kappa$ fixed ($\kappa=\{0.75,1.0\}$). As a guide to the eye, the dashed profile corresponds to the no-memory profile -see Eq. (\ref{['eq: correlation profile no-mem']})-. The parameters we use for the coarse-graining are $N_0=700$, $m=0.9995$ and $k_f^*=200$. The maximum value of $\alpha$ we display (respectively $\alpha=0.7$ and $\alpha=0.975$) corresponds to the last value for which the saddle point of the connected free energy exists.
• Figure 3: Evolution of the correlation length $\xi_k$ as a function of $\alpha$. Four threshold values are displayed: $\kappa\in\{0.5,0.75,1.0,1.25\}$. More particularly, we plot $\xi_k$ for five positions in the path: at the edges ($k=\{0,k_f^*\}$), in the middle ($k=k_f^*/2$), and in the first and last quarters ($k=\{k_f^*/4,3k_f^*/4\}$). In dashed black, we indicate the correlation length for no-memory minima. This plot highlights the translation invariance in the correlation profile, as $\xi_k$ appears to be independent of the position $k$. It also illustrates the divergence of the correlation length at the connected transition (i.e, for $\alpha=\alpha_{\rm connected}$). When $\alpha>\alpha_{\rm connected}$, the free energy does not admit any saddle point.
• Figure 4: Evolution of the averaged correlation profile ${\bf Q}^{*,{\rm avg}}_{k,k'}$ (averaged along the path) as the classification task becomes more difficult. In this case, we increase $\alpha$ while keeping $\kappa$ fixed ($\kappa=\{0.75,1.0\}$). To highlight the exponential trend, the correlation function is plotted in log-linear scale. With this scaling, the slope of the curves corresponds to the correlation length $\xi$. The parameters for the coarse-graining are $N_0=700$, $m=0.9995$ and $k_f^*=200$. The dashed line corresponds to the no-memory profile.
• Figure 5: Margin distribution for the typical connected minima. For each case ($\alpha=\{0.4,0.85,0.975\}$ with fixed $\kappa=1.0$), we make the distinction between minima at the edges of the manifold (in blue) and minima in the core (in orange). In more detail, the edge distribution is obtained by evaluating $\langle \delta(w\!-\!w^{k=0})\rangle_{{\rm 1D},E}$ and the core distribution by evaluating $\langle \delta(w\!-\!w^{k=k_f^*/2})\rangle_{{\rm 1D},E}$; where $\langle \cdot\rangle_{{\rm 1D},E}$ corresponds to the margin measure introduced in Eq. (\ref{['eq: Z energy']}). As a guide to the eye, we also display the margin distribution for typical minima (full line) and for minima at the edges (dashed line) and in the core (dash-dotted line) of the no-memory manifold barbier2025findingrightpathstatistical.