Neuron Block Dynamics for XOR Classification with Zero-Margin

TL;DR

The paper tackles zero-margin nonlinear classification for Gaussian XOR, where standard margin arguments fail due to boundary-proximate data. It introduces a neuron-block dynamics framework in which neurons rapidly organize into four balanced directions and whose collective block signals drive learning. By an two-phase analysis—Phase I with near-independent dynamics and Phase II with block-level growth governed by an average-margin statistic—the authors prove that a two-layer ReLU network trained with vanilla SGD can learn XOR on Gaussian inputs, achieving near-optimal sample complexity and generalization away from the boundary. The results provide a new analytical lens for feature learning under zero-margin conditions and offer practical insights into the robustness of block-level learning beyond Gaussian inputs.

Abstract

The ability of neural networks to learn useful features through stochastic gradient descent (SGD) is a cornerstone of their success. Most theoretical analyses focus on regression or on classification tasks with a positive margin, where worst-case gradient bounds suffice. In contrast, we study zero-margin nonlinear classification by analyzing the Gaussian XOR problem, where inputs are Gaussian and the XOR decision boundary determines labels. In this setting, a non-negligible fraction of data lies arbitrarily close to the boundary, breaking standard margin-based arguments. Building on Glasgow's (2024) analysis, we extend the study of training dynamics from discrete to Gaussian inputs and develop a framework for the dynamics of neuron blocks. We show that neurons cluster into four directions and that block-level signals evolve coherently, a phenomenon essential in the Gaussian setting where individual neuron signals vary significantly. Leveraging this block perspective, we analyze generalization without relying on margin assumptions, adopting an average-case view that distinguishes regions of reliable prediction from regions of persistent error. Numerical experiments confirm the predicted two-phase block dynamics and demonstrate their robustness beyond the Gaussian setting.

Figures (9)

• Figure 1: (a) t-SNE plot of MNIST digits 4 and 9, which overlap substantially. (b) Gaussian XOR setup: $f^*(x) = -\mathrm{sgn}(x_1 x_2)$ with zero margin, since the Gaussian distribution crosses the coordinate axes.
• Figure 2: Neuron dynamics in Phase I. Circles show projections of weights onto $\mathrm{span}(e_1,e_2)$; colors indicate the sign of $a$. Neurons align with $\pm\mu_1,\pm\mu_2$ (green), and blocks remain balanced across quadrants.
• Figure 3: Overview of phases and neuron dynamics. Thicker-bordered boxes emphasize the key dynamics that capture how the network evolves in each phase.
• Figure 4: Neuron dynamics in Phase II.
• Figure 5: Simulated neuron dynamics. (a–c) Weight projections (dots; colors indicate assigned labels) and decision boundaries. The weights progressively align with the directions $\pm\mu_i$. (d) The block mass $N_i^\pm$ vs time (purple: residual mass $\mathcal{R}$). Block masses remain approximately balanced.

Theorems & Definitions (66)

• Theorem 1
• Remark 1: Sample complexity
• Remark 2: Comparison with Boolean XOR
• Corollary 1
• Lemma 1: Phase Ia dynamics
• Lemma 2: Signal-based individual dynamics
• Definition 1: Neuron blocks
• Lemma 3: Pre-balanced blocks
• Definition 2: Signal-heavy network
• Lemma 4: Stability of signal-heavy network