A Survey of Lottery Ticket Hypothesis
Bohan Liu, Zijie Zhang, Peixiong He, Zhensen Wang, Yang Xiao, Ruimeng Ye, Yang Zhou, Wei-Shinn Ku, Bo Hui
TL;DR
The paper surveys the Lottery Ticket Hypothesis (LTH), which posits that dense networks contain sparse winning tickets that can match or exceed dense performance when trained in isolation. It categorizes literature into theory, special models, experimental insights, algorithms, efficiency, relations to other topics, applications, and open issues, and highlights practical challenges such as computation costs and hardware deployment. The authors synthesize methods (e.g., iterative magnitude pruning, rewinding, Early Bird tickets) and model variants (GNNs, Transformers, GANs/VAEs, diffusion models), providing a benchmarking perspective and pointing to open research directions including acceleration, theory, and diffusion-model pruning. Overall, the survey aims to guide researchers toward reproducible experiments, efficient ticket discovery, and broader applicability of LTH across domains.
Abstract
The Lottery Ticket Hypothesis (LTH) states that a dense neural network model contains a highly sparse subnetwork (i.e., winning tickets) that can achieve even better performance than the original model when trained in isolation. While LTH has been proved both empirically and theoretically in many works, there still are some open issues, such as efficiency and scalability, to be addressed. Also, the lack of open-source frameworks and consensual experimental setting poses a challenge to future research on LTH. We, for the first time, examine previous research and studies on LTH from different perspectives. We also discuss issues in existing works and list potential directions for further exploration. This survey aims to provide an in-depth look at the state of LTH and develop a duly maintained platform to conduct experiments and compare with the most updated baselines.