SpinalNet: Deep Neural Network with Gradual Input

TL;DR

SpinalNet introduces a gradual-input neural architecture inspired by the spinal cord, splitting each layer into input, intermediate, and output components to reduce weights and computation while preserving or improving accuracy. The approach is supported by a theoretical basis for universal approximation and a transferred initialization strategy that leverages pre-trained backbones to boost performance on data-rich and data-scarce tasks. Empirical results show competitive or state-of-the-art performance across regression and diverse image-classification benchmarks, with notable gains on MNIST variants and color-image datasets when using VGG backbones or pre-trained networks. The work suggests SpinalNet as a versatile drop-in replacement for fully connected layers, with strong potential for deeper architectures and ensemble methods in practical applications.

Abstract

Deep neural networks (DNNs) have achieved the state of the art performance in numerous fields. However, DNNs need high computation times, and people always expect better performance in a lower computation. Therefore, we study the human somatosensory system and design a neural network (SpinalNet) to achieve higher accuracy with fewer computations. Hidden layers in traditional NNs receive inputs in the previous layer, apply activation function, and then transfer the outcomes to the next layer. In the proposed SpinalNet, each layer is split into three splits: 1) input split, 2) intermediate split, and 3) output split. Input split of each layer receives a part of the inputs. The intermediate split of each layer receives outputs of the intermediate split of the previous layer and outputs of the input split of the current layer. The number of incoming weights becomes significantly lower than traditional DNNs. The SpinalNet can also be used as the fully connected or classification layer of DNN and supports both traditional learning and transfer learning. We observe significant error reductions with lower computational costs in most of the DNNs. Traditional learning on the VGG-5 network with SpinalNet classification layers provided the state-of-the-art (SOTA) performance on QMNIST, Kuzushiji-MNIST, EMNIST (Letters, Digits, and Balanced) datasets. Traditional learning with ImageNet pre-trained initial weights and SpinalNet classification layers provided the SOTA performance on STL-10, Fruits 360, Bird225, and Caltech-101 datasets. The scripts of the proposed SpinalNet are available at the following link: https://github.com/dipuk0506/SpinalNet

Figures (4)

• Figure 1: We develop SpinalNet by mimicking several characteristics of the human somatosensory system to receive large input efficiently and to achieve better performance. (a) How our spinal cord is connected to our body for receiving and sending sensory signals from our body. (b) Structure of the proposed SpinalNet. Each layer of the proposed NN is split into input split, intermediate split, and output split. Each intermediate split receives a portion of the input. All intermediate splits except the intermediate split of the first layer also receive outputs of the previous intermediate split. The output spit adds the weighted outputs of all intermediate splits. The user can also construct and train a SpinalNet for any arbitrary number of inputs, intermediate neurons, and outputs.
• Figure 2: A portion of the human nerve plexus; known as the brachial plexus. The information of any touch or pain reaches the brain through the nerve plexus and the spinal cord. The nerve plexus is a network of intersecting nerves. Our spinal cord receives information gradually. Here, C5-C8 and T1 are vertebrae chang2019vascularized in the human skeleton.
• Figure 3: The visual proof of the universal approximation of the SpinalNet. A simplified version of SpinalNet in (a) can act as a NN of a single hidden layer, drawn in (b). Similarly, a 4 layer SpinalNet in (d) can be equivalent to a NN of one hidden layer (HL), containing four neurons, shown in (c).
• Figure 4: Any traditional hidden layer can be converted to a spinal hidden layer. The traditional hidden layer in (a) is converted to a spinal hidden layer in (b). A spinal hidden layer has the structure of the proposed SpinalNet.