Playing the Lottery With Concave Regularizers for Sparse Trainable Neural Networks

TL;DR

This paper tackles the challenge of finding sparse, trainable subnetworks under the lottery ticket framework by introducing a relaxed binary mask $m\in[0,1]^d$ that is jointly optimized with network weights $\theta$ through a concave regularizer $\mathtt{R}(m)$ in the objective $\mathcal{L}(x; m\odot\theta) + \lambda\mathtt{R}(m)$. It analyzes the approach theoretically in convex-loss settings, deriving bounds on the distance between the learned mask $m^{\star}$ and the target mask $\widetilde{m}$ for both $\ell_1$ and strictly concave (e.g., logarithmic) regularizers, and demonstrates that strictly concave regularization can yield tighter recovery guarantees. Empirically, the method achieves competitive matching-ticket performance with IMP and often surpasses it at higher sparsities across multiple architectures and datasets (e.g., CIFAR/Tiny ImageNet, ResNet/VGG/WideResNet), while providing insights from ablation studies about thresholding and reparameterization. The work advances sparse-training practice by enabling softer pruning decisions via a differentiable-like mask, potentially reducing training cost and enabling efficient, sparse deployment without sacrificing accuracy.

Abstract

The design of sparse neural networks, i.e., of networks with a reduced number of parameters, has been attracting increasing research attention in the last few years. The use of sparse models may significantly reduce the computational and storage footprint in the inference phase. In this context, the lottery ticket hypothesis (LTH) constitutes a breakthrough result, that addresses not only the performance of the inference phase, but also of the training phase. It states that it is possible to extract effective sparse subnetworks, called winning tickets, that can be trained in isolation. The development of effective methods to play the lottery, i.e., to find winning tickets, is still an open problem. In this article, we propose a novel class of methods to play the lottery. The key point is the use of concave regularization to promote the sparsity of a relaxed binary mask, which represents the network topology. We theoretically analyze the effectiveness of the proposed method in the convex framework. Then, we propose extended numerical tests on various datasets and architectures, that show that the proposed method can improve the performance of state-of-the-art algorithms.

Figures (9)

• Figure 1: Three-stage pipeline for pruning. The initialization of the retraining stage is done either with fine-tuning or rewinding.
• Figure 2: Eight samples from the MNIST dataset
• Figure 3: Binary classification on MNIST dataset with logistic regression (LG): accuracy and sparsity with respect to the design parameter $\lambda$.
• Figure 4: (a) An 8-sparse mask obtained with logarithmic regularization; (c) Samples of Fig. \ref{['fig:mnist']} masked via (b). We can see that 8 pixels may be sufficient to distinguish between digits 0 and 1.
• Figure 5: Test accuracy and sparsity of the subnetworks of ResNet-20 produced by IMP and the proposed method on CIFAR-10, CIFAR-100, and Tiny ImageNet.

Theorems & Definitions (3)

• Theorem 1
• Theorem 2
• Lemma 1