The Symmetric Perceptron: a Teacher-Student Scenario

Abstract

We introduce and solve a teacher-student formulation of the symmetric binary Perceptron, turning a traditionally storage-oriented model into a planted inference problem with a guaranteed solution at any sample density. We adapt the formulation of the symmetric Perceptron which traditionally considers either the u-shaped potential or the rectangular one, by including labels in both regions. With this formulation, we analyze both the Bayes-optimal regime at for noise-less examples and the effect of thermal noise under two different potential/classification rules. Using annealed and quenched free-entropy calculations in the high-dimensional limit, we map the phase diagram in the three control parameters, namely the sample density $α$, the distance between the origin and one of the symmetric hyperplanes $κ$ and temperature $T$, and identify a robust scenario where learning is organized by a second-order instability that creates teacher-correlated suboptimal states, followed by a first-order transition to full alignment. We show how this structure depends on the choice of potential, the interplay between metastability of the suboptimal solution and its melting towards the planted configuration, which is relevant for Monte Carlo-based optimization algorithms.

Figures (6)

• Figure 1: Decision boundary for a symmetric Perceptron in $N=2$ with $w=\left[-1 \slash \sqrt{2}, 1 \slash \sqrt{2}\right]$ and $\kappa=1$, illustrating a case where the data in the red region are labelled $+1$, while the ones in the blue region are labeled $-1$.
• Figure 2: Left: the potential $V^{(0)}$ for both labels $\sigma_0 = \pm 1$. With this potential, any error has a fixed cost. Right: the linear potential $V^{(1)}$ for both labels $\sigma_0 = \pm 1$. This potential tends to penalize more errors that are far away from the decision boundary.
• Figure 3: Snapshots of free energy profiles at $6$ different values of $\kappa$, corresponding to the thin vertical lines in Fig.\ref{['fig:PD_bothAnnealed_andQuenched']} (left panel). Each panel (a)$\to$(f) shows several free energy profiles $f(R)$ at different values of $\alpha$. The thicker curves correspond to the values of the first/second order transition and the spinodal (when different). In blue we illustrate first order transitions, in red second order ones and in yellow the melting (spinodal) point of the sub-optimal solution $0<R<1$. The colors used here are the same as in Fig. \ref{['fig:PD_bothAnnealed_andQuenched']}.
• Figure 4: Left: Phase diagram of the symmetric binary Perceptron at $T=0$, using the annealed approximation. Right: Phase diagram of the symmetric binary Perceptron at $T=0$, using the quenched approximation in the Bayes-Optimal setting. The vertical lines indicate the values of $\kappa$ used on Fig. \ref{['fig:Free_energies_Annealed_T0']} (resp. Fig. \ref{['fig:Free_energies_Quenched_T0XXX']}) for the annealed (resp. quenched) case. In both cases, the blue line indicated the first order transition, the red line the second order one and the yellow line the spinodal, that is when the sub-optimal solution disappear. In the range of $\kappa$ where the second order phase transition occurs, the dashed-blue line only corresponding to when the unstable paramagnetic state and the teacher state have the same free energy.
• Figure 5: Snapshots of free energy profiles at 6 different values of $\kappa$ at $T=0$ in the Bayes-Optimal setting. We can observe a phase diagram that is qualitatively similar to the annealed case. From small values of $\kappa$, we observe first a first order phase transition, follow by a melting toward the teacher state at higher values of $\alpha$. Then, we have the at larger $\kappa$ a second order phase transition before the melting.