Robustness in sparse artificial neural networks trained with adaptive topology

TL;DR

The robustness of sparse artificial neural networks trained with adaptive topology is investigated, exploring their performance under various perturbations including random link removal, adversarial attack, and link weight shuffling.

Abstract

We investigate the robustness of sparse artificial neural networks trained with adaptive topology. We focus on a simple yet effective architecture consisting of three sparse layers with 99% sparsity followed by a dense layer, applied to image classification tasks such as MNIST and Fashion MNIST. By updating the topology of the sparse layers between each epoch, we achieve competitive accuracy despite the significantly reduced number of weights. Our primary contribution is a detailed analysis of the robustness of these networks, exploring their performance under various perturbations including random link removal, adversarial attack, and link weight shuffling. Through extensive experiments, we demonstrate that adaptive topology not only enhances efficiency but also maintains robustness. This work highlights the potential of adaptive sparse networks as a promising direction for developing efficient and reliable deep learning models.

Figures (7)

• Figure 1: Illustration of the network architecture. The input layer (shown on the left) contains 28 x 28 pixels, followed by 3 sparsely connected neuron layers with 1000 neurons each. Blue coloured connections have a positive link weight, whereas red colour indicates a negative weight. The last layer, providing the readout is densely connected to the third sparse layer.
• Figure 2: Duration of the topology update. We show the bar chart of the average duration in seconds for a single topology update over 4 training instances and 50 epochs. The results for the CH3L3 method are shown in light blue, whereas for the Random Link Regrowth method in purple, where the dataset is marked under the corresponding bars.
• Figure 3: Accuracy during training. We show the accuracy ( fraction of correctly classified inputs, where the shaded area indicates the standard deviation) as a function of the number of epochs (on logarithmic scale). The results for the CH3L3 method are shown in blue, for the Random Link Regrowth method in red. Results are displayed for the following datasets: a) MNIST, b) FashionMNIST, c) KMNIST, and d) EMNIST Letters. Each setup was run 10 times with a different random seed and each training instance run for 1000 epochs with a topology update between consecutive epochs.
• Figure 4: Robustness of trained sparse networks against link removal. Average accuracy over 32 networks as a function of the fraction of removed links (perturbation value) for systems trained according to the CHCL3 method (continous lines) and the RLR method (dashed lines) for a) the MNIST dataset, b) the Fashion MNIST dataset, c) the KMNIST dataset, and d) the EMNIST letters dataset. Results for Reverse Weight Order Pruning are shown in red, for Random Pruning in green and for Weight Order Pruning in gray.
• Figure 5: Robustness of trained sparse networks against perturbing the weights by shuffling or adding noise. Accuracy as a function of the perturbation value for networks trained with the CH3L3 Link Regrowth method (continuous lines) and with the RLR method (dashed lines) for a) the MNIST dataset, b) the Fashion MNIST dataset, c) the KMNIST dataset, and d) the EMNIST letters dataset. Results for Weight Shuffling are shown in purple, for Weight Modification in light blue.