SuperTickets: Drawing Task-Agnostic Lottery Tickets from Supernets via Jointly Architecture Searching and Parameter Pruning

TL;DR

The paper tackles the high cost of NAS and pruning pipelines by proposing SuperTickets, a two-in-one training framework that simultaneously searches for efficient architectures and prunes weights directly from a task-agnostic supernet. It introduces progressive pruning and iterative reactivation to dynamically adapt subnetworks during training, enabling better accuracy-efficiency trade-offs and potential deployment without retraining. The work demonstrates that SuperTickets exist within supernets and can transfer across datasets and tasks, validated through extensive experiments on image classification, segmentation, and pose estimation across four datasets. This approach promises substantial reductions in compute while maintaining high performance, advancing practical, task-agnostic neural network design.

Abstract

Neural architecture search (NAS) has demonstrated amazing success in searching for efficient deep neural networks (DNNs) from a given supernet. In parallel, the lottery ticket hypothesis has shown that DNNs contain small subnetworks that can be trained from scratch to achieve a comparable or higher accuracy than original DNNs. As such, it is currently a common practice to develop efficient DNNs via a pipeline of first search and then prune. Nevertheless, doing so often requires a search-train-prune-retrain process and thus prohibitive computational cost. In this paper, we discover for the first time that both efficient DNNs and their lottery subnetworks (i.e., lottery tickets) can be directly identified from a supernet, which we term as SuperTickets, via a two-in-one training scheme with jointly architecture searching and parameter pruning. Moreover, we develop a progressive and unified SuperTickets identification strategy that allows the connectivity of subnetworks to change during supernet training, achieving better accuracy and efficiency trade-offs than conventional sparse training. Finally, we evaluate whether such identified SuperTickets drawn from one task can transfer well to other tasks, validating their potential of handling multiple tasks simultaneously. Extensive experiments and ablation studies on three tasks and four benchmark datasets validate that our proposed SuperTickets achieve boosted accuracy and efficiency trade-offs than both typical NAS and pruning pipelines, regardless of having retraining or not. Codes and pretrained models are available at https://github.com/RICE-EIC/SuperTickets.

Figures (13)

• Figure 1: Illustrating first-search-then-prune (S+P) vs. our two-in-One training.
• Figure 2: Comparing the mIoU, mAcc, aAcc and inference FLOPs of the resulting networks from the proposed two-in-one training and first-search-then-prune (S+P) baselines on semantic segmentation task and Cityscapes dataset, where Rand., Mag., and Grad. represent random, magnitude, and graident-based pruning, respectively. Note that each method has a series of points for representing different pruning ratios ranging from 10% to 98%. All accuracies are averaged over three runs.
• Figure 3: Comparing the AP, AP$^M$, AP$^L$ and inference FLOPs of the resulting networks from the proposed two-in-one training and baselines on human pose estimation task and COCO keypoint dataset. Each method has a series of points for representing different pruning ratios ranging from 10% to 98%. All accuracies are averaged over three runs.
• Figure 4: Comparing the top-1/5 accuracy and FLOPs of the proposed SuperTickets and S+P baselines on ImageNet. Each method has a series of points to represent different pruning ratios ranging from 10% to 98%. All accuracies are averaged over three runs. We also benchmark all methods with retraining (denoted as w/ RT).
• Figure 5: Comparing the mIoU, mAcc, aAcc and inference FLOPs of the proposed SuperTickets and S+P baselines on Cityscapes and ADE20K datasets. Each method has a series of points to represent different pruning ratios ranging from 10% to 98%.