Residual Random Neural Networks
M. Andrecut
TL;DR
This work challenges the standard view that random-SLFNNs require a hidden layer much larger than data dimensionality, showing that high data dimensionality $M$ enables good accuracy with $J$ on the order of $M$. It introduces Residual Random Neural Networks (RRNN) and their kernel analogue RRKN, which iteratively correct residual errors using new random projections or kernels, achieving competitive results on MNIST and fashion-MNIST, e.g., MNIST around $99\%$ accuracy and fMNIST around $91\%$. The approach leverages high-dimensional geometry (volume concentration and almost-orthogonality) to justify improved separability, and extends to an encryption scheme using orthonormal matrices for protecting data and models. Overall, the paper provides a scalable, low-complexity training paradigm with practical security properties for random projection-based learners and their kernel extensions.
Abstract
The single-layer feedforward neural network with random weights is a recurring motif in the neural networks literature. The advantage of these networks is their simplified training, which reduces to solving a ridge-regression problem. A general assumption is that these networks require a large number of hidden neurons relative to the dimensionality of the data samples, in order to achieve good classification accuracy. Contrary to this assumption, here we show that one can obtain good classification results even if the number of hidden neurons has the same order of magnitude as the dimensionality of the data samples, if this dimensionality is reasonably high. Inspired by this result, we also develop an efficient iterative residual training method for such random neural networks, and we extend the algorithm to the least-squares kernel version of the neural network model. Moreover, we also describe an encryption (obfuscation) method which can be used to protect both the data and the resulted network model.