Pruning as Evolution: Emergent Sparsity Through Selection Dynamics in Neural Networks

TL;DR

Pruning is reframed as an emergent property of learning dynamics, where parameter groups act as competing populations with masses $p_i(t)$ that evolve under local fitness signals $\phi_i$. Pruning corresponds to population extinction when $p_i(t)\to 0$, yielding an accuracy–sparsity tradeoff without explicit pruning schedules. The authors formalize three evolutionary dynamics—replicator, normalized growth, and selection–mutation—and demonstrate their applicability on MNIST with a population-scaled MLP, showing meaningful sparsity can be achieved post-training while preserving most performance. This process-level perspective aligns with the decentralized, stochastic nature of gradient-based learning and provides a principled link to existing pruning heuristics, while pointing to future extensions in redundancy-aware fitness and scalable architectures.

Abstract

Neural networks are commonly trained in highly overparameterized regimes, yet empirical evidence consistently shows that many parameters become redundant during learning. Most existing pruning approaches impose sparsity through explicit intervention, such as importance-based thresholding or regularization penalties, implicitly treating pruning as a centralized decision applied to a trained model. This assumption is misaligned with the decentralized, stochastic, and path-dependent character of gradient-based training. We propose an evolutionary perspective on pruning: parameter groups (neurons, filters, heads) are modeled as populations whose influence evolves continuously under selection pressure. Under this view, pruning corresponds to population extinction: components with persistently low fitness gradually lose influence and can be removed without discrete pruning schedules and without requiring equilibrium computation. We formalize neural pruning as an evolutionary process over population masses, derive selection dynamics governing mass evolution, and connect fitness to local learning signals. We validate the framework on MNIST using a population-scaled MLP (784--512--256--10) with 768 prunable neuron populations. All dynamics reach dense baselines near 98\% test accuracy. We benchmark post-training hard pruning at target sparsity levels (35--50\%): pruning 35\% yields $\approx$95.5\% test accuracy, while pruning 50\% yields $\approx$88.3--88.6\%, depending on the dynamic. These results demonstrate that evolutionary selection produces a measurable accuracy--sparsity tradeoff without explicit pruning schedules during training.