Pruning as a Game: Equilibrium-Driven Sparsification of Neural Networks
Zubair Shah, Noaman Khan
TL;DR
Pruning is addressed as an equilibrium outcome of strategic interaction among overparameterized neural-network parameter groups, modeled as players with continuous participation variables $s_i\in[0,1]$. The authors show sparsity emerges when continued participation becomes a dominated strategy at a Nash equilibrium, and they derive a simple equilibrium-driven pruning algorithm that jointly updates $\theta$ and $s$ via gradient descent and projected ascent using a utility $U_i(s_i,s_{-i})=B_i-C_i$, where $B_i$ captures marginal contribution $\alpha\,s_i\langle \nabla_{\theta_i}\mathcal{L},\theta_i\rangle$ and $C_i$ includes an elastic-net-like penalty plus a redundancy term. Theoretical analysis yields best-response conditions and explicit sparsity criteria, while experiments on MNIST with an MLP demonstrate competitive sparsity–accuracy trade-offs and a bimodal distribution of final participation values, supporting the equilibrium interpretation. This work provides a principled framework that connects existing pruning heuristics to equilibrium dynamics and offers a scalable, end-to-end pruning strategy with potential extensions to structured pruning and larger models.
Abstract
Neural network pruning is widely used to reduce model size and computational cost. Yet, most existing methods treat sparsity as an externally imposed constraint, enforced through heuristic importance scores or training-time regularization. In this work, we propose a fundamentally different perspective: pruning as an equilibrium outcome of strategic interaction among model components. We model parameter groups such as weights, neurons, or filters as players in a continuous non-cooperative game, where each player selects its level of participation in the network to balance contribution against redundancy and competition. Within this formulation, sparsity emerges naturally when continued participation becomes a dominated strategy at equilibrium. We analyze the resulting game and show that dominated players collapse to zero participation under mild conditions, providing a principled explanation for pruning behavior. Building on this insight, we derive a simple equilibrium-driven pruning algorithm that jointly updates network parameters and participation variables without relying on explicit importance scores. This work focuses on establishing a principled formulation and empirical validation of pruning as an equilibrium phenomenon, rather than exhaustive architectural or large-scale benchmarking. Experiments on standard benchmarks demonstrate that the proposed approach achieves competitive sparsity-accuracy trade-offs while offering an interpretable, theory-grounded alternative to existing pruning methods.