Emergence of Structure in Ensembles of Random Neural Networks

TL;DR

The paper investigates how ensembles of random classifiers can exhibit deterministic, near-optimal behavior without traditional training by weighting random perceptrons with a Gibbs measure over the loss. It develops a training-free framework, derives an analytical expression for the optimal inverse temperature $\beta^*$ in the Gaussian-data setting (showing $\beta^* = \pi\sqrt{d-2}$), and demonstrates universality with respect to the teacher and the ensemble size. Empirical results on MNIST reveal a comparable minimum near $\beta \approx 20$, with the ensemble generalizing to unseen data and consistent $\beta^*$ across multiple tasks, supporting the universality claim beyond isotropic data. The work also offers a physical interpretation as self-organization in a large, non-interacting ensemble and discusses hardware-friendly inference and future extensions to richer random architectures and noisy measurements, highlighting potential practical impact for scalable, low-trainingInference pipelines.

Abstract

Randomness is ubiquitous in many applications across data science and machine learning. Remarkably, systems composed of random components often display emergent global behaviors that appear deterministic, manifesting a transition from microscopic disorder to macroscopic organization. In this work, we introduce a theoretical model for studying the emergence of collective behaviors in ensembles of random classifiers. We argue that, if the ensemble is weighted through the Gibbs measure defined by adopting the classification loss as an energy, then there exists a finite temperature parameter for the distribution such that the classification is optimal, with respect to the loss (or the energy). Interestingly, for the case in which samples are generated by a Gaussian distribution and labels are constructed by employing a teacher perceptron, we analytically prove and numerically confirm that such optimal temperature does not depend neither on the teacher classifier (which is, by construction of the learning problem, unknown), nor on the number of random classifiers, highlighting the universal nature of the observed behavior. Experiments on the MNIST dataset underline the relevance of this phenomenon in high-quality, noiseless, datasets. Finally, a physical analogy allows us to shed light on the self-organizing nature of the studied phenomenon.

Figures (10)

• Figure 1: Model of the proposed architecture, with null biases. The gray weights are sampled from a normal distribution, while red weights are computed analytically by the rule described in Equation \ref{['eq:rule']}. The highlighted subnetwork represents the flow of information from the input, to one randomly constructed classifier and to the output. The hidden layer can be considered as an ensemble of random elementary neural networks (perceptrons) whose output is then combined through the analytically computed weights, which do not require a training phase to be estimated, with each weight in the last layer being independent from the others. Both hidden and output neurons adopt the $\operatorname{sign}$ activation function.
• Figure 2: Distribution of the composition of the loss function and the random variable $\mathbf w \sim \mathcal{N}(\mathbf 0, \mathbf I_d)$, as the dimensions increase it becomes exponentially harder to sample classifier that significantly outperform the average case
• Figure 3: Loss of the ensemble for different values of $\beta$. The profile of the function shows a clear structure with a distinct minimmum, which significantly overcomes the concentration of the loss in high dimensional data.
• Figure 4: Structure of the loss profiles in function of perturbations of $d$, $n$ and $N$. The color scale indicates an increase of the fixed value from blue to red. The position of the optimal inverse temperature appears to be influenced solely by $d$
• Figure 5: Profile of the loss for different teachers classifiers. Despite of the fluctuations, the position of the minimum is virtually constant among the different simulations