A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations

TL;DR

This survey provides a structured taxonomy and comprehensive evaluation of deep neural network pruning, covering universal versus specific speedups, timing of pruning, and pruning-by-criteria versus learning-based approaches. It analyzes eight contrast settings through extensive experiments, including unstructured, structured, and semi-structured pruning, across CNNs, Transformers, and large language models, and highlights run-time dynamics and transferability. Key contributions include a taxonomy, a large empirical comparison, a curated resource compilation, and actionable recommendations to guide practitioners in selecting pruning strategies for various hardware, data, and application constraints. The work emphasizes practical impact by detailing guidelines for method choice, data usage, and integration with other compression techniques, and it points to future directions in theory, AutoML/NAS integration, and standardized evaluation benchmarks.

Abstract

Modern deep neural networks, particularly recent large language models, come with massive model sizes that require significant computational and storage resources. To enable the deployment of modern models on resource-constrained environments and accelerate inference time, researchers have increasingly explored pruning techniques as a popular research direction in neural network compression. However, there is a dearth of up-to-date comprehensive review papers on pruning. To address this issue, in this survey, we provide a comprehensive review of existing research works on deep neural network pruning in a taxonomy of 1) universal/specific speedup, 2) when to prune, 3) how to prune, and 4) fusion of pruning and other compression techniques. We then provide a thorough comparative analysis of eight pairs of contrast settings for pruning and explore emerging topics, including pruning for large language models, large multimodal models, post-training pruning, and different supervision levels for pruning to shed light on the commonalities and differences of existing methods and lay the foundation for further method development. To facilitate future research, we build a curated collection of datasets, networks, and evaluations on different applications. Finally, we provide valuable recommendations on selecting pruning methods and prospect several promising research directions. We build a repository at https://github.com/hrcheng1066/awesome-pruning.

Figures (12)

• Figure 1: The number of papers on neural network pruning and compression from 1989-2023.
• Figure 2: An overview of the hierarchical structure of the survey, with sections involving large models highlighted by a pink triangle.
• Figure 3: The visualization of unstructured pruning. The light orange circles denote neurons.
• Figure 4: The visualization of typical structured and semi-structured pruning: each Conv. consists of cubes representing weights. Orange cube sets/blocks indicate the pruned structures. Best view in color and zoom in.
• Figure 5: The typical pipelines of static pruning. Dashed boxes represent models and solid boxes denote actions. Dashed arrows indicate optional steps.