Learning a Sparse Neural Network using IHT
Saeed Damadi, Soroush Zolfaghari, Mahdi Rezaie, Jinglai Shen
TL;DR
This work applies Iterative Hard Thresholding (IHT) theory to train sparse neural networks by framing sparsity as a constrained optimization with ||θ||_0 ≤ s and an ε-optimality condition comparing dense and sparse losses. It leverages Restricted Strong Smoothness (RSS) concepts to establish convergence of IHT to HT-stable stationary points, and introduces a Monte Carlo procedure to estimate the RSS constant L_{2s} for neural nets to guide the learning rate. Empirically, on a single-layer NN trained with the IRIS dataset, sparse networks with as few as roughly 5 nonzero parameters achieve comparable loss and accuracy to the dense model, validating the ε-optimality claim and demonstrating practical sparsification benefits. The results bridge sparse optimization theory with NN sparsification, offering concrete steps for rate selection and HT-stability verification, and setting the stage for extending to deeper architectures.
Abstract
The core of a good model is in its ability to focus only on important information that reflects the basic patterns and consistencies, thus pulling out a clear, noise-free signal from the dataset. This necessitates using a simplified model defined by fewer parameters. The importance of theoretical foundations becomes clear in this context, as this paper relies on established results from the domain of advanced sparse optimization, particularly those addressing nonlinear differentiable functions. The need for such theoretical foundations is further highlighted by the trend that as computational power for training NNs increases, so does the complexity of the models in terms of a higher number of parameters. In practical scenarios, these large models are often simplified to more manageable versions with fewer parameters. Understanding why these simplified models with less number of parameters remain effective raises a crucial question. Understanding why these simplified models with fewer parameters remain effective raises an important question. This leads to the broader question of whether there is a theoretical framework that can clearly explain these empirical observations. Recent developments, such as establishing necessary conditions for the convergence of iterative hard thresholding (IHT) to a sparse local minimum (a sparse method analogous to gradient descent) are promising. The remarkable capacity of the IHT algorithm to accurately identify and learn the locations of nonzero parameters underscores its practical effectiveness and utility. This paper aims to investigate whether the theoretical prerequisites for such convergence are applicable in the realm of neural network (NN) training by providing justification for all the necessary conditions for convergence. Then, these conditions are validated by experiments on a single-layer NN, using the IRIS dataset as a testbed.