On the Sparsity of the Strong Lottery Ticket Hypothesis

TL;DR

The first proof of the SLTH in classical settings, such as dense and equivariant networks, with guarantees on the sparsity of the subnetworks is provided, central to the proof of an essentially tight bound on the Random Fixed-Size Subset Sum Problem (RFSS), a variant of the RSS Problem in which they only ask for subsets of a given size.

Abstract

Considerable research efforts have recently been made to show that a random neural network $N$ contains subnetworks capable of accurately approximating any given neural network that is sufficiently smaller than $N$, without any training. This line of research, known as the Strong Lottery Ticket Hypothesis (SLTH), was originally motivated by the weaker Lottery Ticket Hypothesis, which states that a sufficiently large random neural network $N$ contains \emph{sparse} subnetworks that can be trained efficiently to achieve performance comparable to that of training the entire network $N$. Despite its original motivation, results on the SLTH have so far not provided any guarantee on the size of subnetworks. Such limitation is due to the nature of the main technical tool leveraged by these results, the Random Subset Sum (RSS) Problem. Informally, the RSS Problem asks how large a random i.i.d. sample $Ω$ should be so that we are able to approximate any number in $[-1,1]$, up to an error of $ ε$, as the sum of a suitable subset of $Ω$. We provide the first proof of the SLTH in classical settings, such as dense and equivariant networks, with guarantees on the sparsity of the subnetworks. Central to our results, is the proof of an essentially tight bound on the Random Fixed-Size Subset Sum Problem (RFSS), a variant of the RSS Problem in which we only ask for subsets of a given size, which is of independent interest.

Figures (3)

• Figure 1: A qualitative plot showing the relationship between the density $\gamma$ of a winning ticket and the overparameterization required by Theorem \ref{['thm:pensia']} for a target network with $m_t$ parameters. Earlier results from Pensia et al. Pensia and Malach et al. malachProvingLotteryTicket2020 are shown for comparison.
• Figure 2: Simplified representation of the procedure for finding Lottery Tickets (LTH). A large random neural network (step 1) is trained by iterative pruning with rewind: when the loss reaches a local minimum (step 2), some weights with smallest absolute value are pruned (step 3) and the value of the remaining edges is then reset to that of the initialization (step 4); finally, training is resumed and the final network is obtained (step 5). Remarkably, the sparser subnetwork is consistently able to reach a loss not larger than that right after pruning.
• Figure 3: Simplified representation of the procedure for finding Strongly Lottery Tickets (SLTH) / Training by pruning. Previous work has shown that it is possible to sparsify large random neural network in order to obtain subnetworks that achieve good performance for a task under consideration, motivating the Strong Lottery Ticket Hypothesis. No training is required.

Theorems & Definitions (22)

• Theorem 1: Informal statement of previous SLTH results
• Definition 1: sum-bounded
• Theorem 2
• Remark
• Lemma 1
• Corollary 1
• proof : Proof Idea.
• Corollary 2
• proof : Proof of Theorem \ref{['thm:srss']}
• Theorem 3: SSLTH for DNNs