The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training

TL;DR

The paper investigates random pruning as a practical baseline for sparse training, arguing that naive, initialization-time random sparsity can match dense performance when networks are large and layer-wise sparsity patterns are suitably chosen. It systematically compares six sparsity schemes (ERK, ERK+, Uniform, Uniform+, SNIP, GraSP), adapts pruning-at-initialization criteria for random masks, and evaluates on CIFAR and ImageNet. Key findings include that larger networks bridge the gap to dense accuracy at high sparsities, ERK-based schemes often outperform more sophisticated criteria, and random pruning yields benefits in OoD detection, uncertainty estimation, and adversarial robustness. The results suggest a broader potential for sparse training at scale and emphasize the practical value of simple, random baselines, with code and reproducibility resources provided.

Abstract

Random pruning is arguably the most naive way to attain sparsity in neural networks, but has been deemed uncompetitive by either post-training pruning or sparse training. In this paper, we focus on sparse training and highlight a perhaps counter-intuitive finding, that random pruning at initialization can be quite powerful for the sparse training of modern neural networks. Without any delicate pruning criteria or carefully pursued sparsity structures, we empirically demonstrate that sparsely training a randomly pruned network from scratch can match the performance of its dense equivalent. There are two key factors that contribute to this revival: (i) the network sizes matter: as the original dense networks grow wider and deeper, the performance of training a randomly pruned sparse network will quickly grow to matching that of its dense equivalent, even at high sparsity ratios; (ii) appropriate layer-wise sparsity ratios can be pre-chosen for sparse training, which shows to be another important performance booster. Simple as it looks, a randomly pruned subnetwork of Wide ResNet-50 can be sparsely trained to outperforming a dense Wide ResNet-50, on ImageNet. We also observed such randomly pruned networks outperform dense counterparts in other favorable aspects, such as out-of-distribution detection, uncertainty estimation, and adversarial robustness. Overall, our results strongly suggest there is larger-than-expected room for sparse training at scale, and the benefits of sparsity might be more universal beyond carefully designed pruning. Our source code can be found at https://github.com/VITA-Group/Random_Pruning.

Figures (17)

• Figure 1: From shallow to deep. Test accuracy of training randomly pruned subnetworks from scratch with different depth on CIFAR-10. ResNet-A refers to a ResNet model with A layers in total.
• Figure 2: From narrow to wide. Test accuracy of training randomly pruned subnetworks from scratch with different width on CIFAR-10. ResNet-A-B refers to a ResNet model with A layers in total and B filters in the first convolutional layer.
• Figure 3: Uncertainty estimation (ECE). The experiments are conducted with various models on CIFAR-10. Lower ECE values represent better uncertainty estimation.
• Figure 4: Out-of-distribution performance (ROC-AUC). Experiments are conducted with models trained on CIFAR-10, tested on CIFAR-100. Higher ROC-AUC refers to better OoD performance.
• Figure 5: Summary of evaluation on ImageNet. Various Evaluation of ResNets on ImageNet, including predictive accuracy on the original ImageNet, adversarial robustness with FGSM, OoD performance on ImageNet-O, and uncertainty (ECE and NLL). The sparsity of randomly pruned subnetworks is set as [0.7, 0.5, 0.3] from left to right for each line.