SPDY: Accurate Pruning with Speedup Guarantees

TL;DR

SPDY tackles the gap between pruning and actual inference speed by formulating speedup-aware unstructured pruning as a constrained optimization solvable by dynamic programming. It couples an efficient DP solver with a learned per-layer error metric, refined through a reconstruction database built from AdaPrune, to produce layer-wise sparsity profiles that meet target speeds with minimal accuracy loss. The approach yields superior speedup-accuracy trade-offs in one-shot and gradual pruning across vision and language models and extends to post-training GPU pruning via global AdaPrune (gAP). Realistic experiments show SPDY often achieves the same speed with lower sparsity and demonstrates robust applicability to CPU and GPU acceleration, with potential extensions to other compression settings. These results offer a practical, scalable framework for speed-aware model compression in diverse deployment scenarios.

Abstract

The recent focus on the efficiency of deep neural networks (DNNs) has led to significant work on model compression approaches, of which weight pruning is one of the most popular. At the same time, there is rapidly-growing computational support for efficiently executing the unstructured-sparse models obtained via pruning. Yet, most existing pruning methods minimize just the number of remaining weights, i.e. the size of the model, rather than optimizing for inference time. We address this gap by introducing SPDY, a new compression method which automatically determines layer-wise sparsity targets achieving a desired inference speedup on a given system, while minimizing accuracy loss. SPDY is composed of two new techniques: the first is an efficient dynamic programming algorithm for solving the speedup-constrained layer-wise compression problem assuming a set of given layer-wise sensitivity scores; the second is a local search procedure for determining accurate layer-wise sensitivity scores. Experiments across popular vision and language models show that SPDY guarantees speedups while recovering higher accuracy relative to existing strategies, both for one-shot and gradual pruning scenarios, and is compatible with most existing pruning approaches. We also extend our approach to the recently-proposed task of pruning with very little data, where we achieve the best known accuracy recovery when pruning to the GPU-supported 2:4 sparsity pattern.

Figures (9)

• Figure 1: Speedup and relative performance measure drop trade-off after gradual pruning for SPDY and baselines on various models.
• Figure 2: Comparison of SPDY with a direct search and genetic programming.
• Figure 2: Comparing profiles when post training pruning ResNet models to the same speed as their next smallest dense counter-part.
• Figure 3: A visual overview of the full SPDY method.
• Figure 4: Comparison with results of speedup-related unstructured pruning methods on ResNet50 and MobileNetV1 in terms of relative accuracy drop.