Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Separate, Dynamic and Differentiable (SMART) Pruner for Block/Output Channel Pruning on Computer Vision Tasks

Guanhua Ding, Zexi Ye, Zhen Zhong, Gang Li, David Shao

TL;DR

SMART pruner tackles block pruning by integrating cross-layer and within-layer weight importance with a differentiable Top-$k$ mask and a dynamic temperature schedule. By reformulating the $L_0$-norm constrained pruning into an unconstrained objective $L(w \odot f_\tau(m))$, and proving equivalence to the original problem (Theorem 3.3 and related results), it enables gradient-based optimization with convergence guarantees as $\tau \to 0$. The approach is complemented by a three-stage training flow and a dynamic temperature trick that escapes non-sparse local minima (Theorem 3.7). Empirically, SMART achieves state-of-the-art block pruning performance across multiple CV tasks and block shapes, with reduced hyperparameter tuning, making it practical for industrial edge deployments that demand accuracy, controllable sparsity, and stable convergence.

Abstract

Block pruning, which eliminates contiguous blocks of weights, is a structural pruning method that can significantly enhance the performance of neural processing units (NPUs). In industrial applications, an ideal block pruning algorithm should meet three key requirements: (1) maintain high accuracy across diverse models and tasks, as machine learning deployments on edge devices are typically accuracy-critical; (2) offer precise control over resource constraints to facilitate user adoption; and (3) provide convergence guarantees to prevent performance instability. However, to the best of our knowledge, no existing block pruning algorithm satisfies all three requirements simultaneously. In this paper, we introduce SMART (Separate, Dynamic, and Differentiable) pruning, a novel algorithm designed to address this gap. SMART leverages both weight and activation information to enhance accuracy, employs a differentiable top-k operator for precise control of resource constraints, and offers convergence guarantees under mild conditions. Extensive experiments involving seven models, four datasets, three different block types, and three computer vision tasks demonstrate that SMART pruning achieves state-of-the-art performance in block pruning.

Separate, Dynamic and Differentiable (SMART) Pruner for Block/Output Channel Pruning on Computer Vision Tasks

TL;DR

SMART pruner tackles block pruning by integrating cross-layer and within-layer weight importance with a differentiable Top- mask and a dynamic temperature schedule. By reformulating the -norm constrained pruning into an unconstrained objective , and proving equivalence to the original problem (Theorem 3.3 and related results), it enables gradient-based optimization with convergence guarantees as . The approach is complemented by a three-stage training flow and a dynamic temperature trick that escapes non-sparse local minima (Theorem 3.7). Empirically, SMART achieves state-of-the-art block pruning performance across multiple CV tasks and block shapes, with reduced hyperparameter tuning, making it practical for industrial edge deployments that demand accuracy, controllable sparsity, and stable convergence.

Abstract

Block pruning, which eliminates contiguous blocks of weights, is a structural pruning method that can significantly enhance the performance of neural processing units (NPUs). In industrial applications, an ideal block pruning algorithm should meet three key requirements: (1) maintain high accuracy across diverse models and tasks, as machine learning deployments on edge devices are typically accuracy-critical; (2) offer precise control over resource constraints to facilitate user adoption; and (3) provide convergence guarantees to prevent performance instability. However, to the best of our knowledge, no existing block pruning algorithm satisfies all three requirements simultaneously. In this paper, we introduce SMART (Separate, Dynamic, and Differentiable) pruning, a novel algorithm designed to address this gap. SMART leverages both weight and activation information to enhance accuracy, employs a differentiable top-k operator for precise control of resource constraints, and offers convergence guarantees under mild conditions. Extensive experiments involving seven models, four datasets, three different block types, and three computer vision tasks demonstrate that SMART pruning achieves state-of-the-art performance in block pruning.
Paper Structure (18 sections, 7 theorems, 27 equations, 3 figures, 5 tables, 2 algorithms)

This paper contains 18 sections, 7 theorems, 27 equations, 3 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Differentiable Top $k$ Operator
  5. SMART pruning algorithm
  6. Dynamic temperature parameter trick
  7. Experimental Results
  8. Ablation Study
  9. Conclusion
  10. Proof of Theorem 3.1
  11. Proof of Proposition 3.2
  12. Proof of Theorem 3.3
  13. Proof of Theorem 3.4
  14. Proof of Theorem 3.7
  15. The Accumulated Weight and Gradient (AWG) Pruning Algorithm
  16. ...and 3 more sections

Key Result

Theorem 3.1

Suppose $N \geq 2$ and $1 \leq k \leq N$. As the temperature parameter $\tau$ approaches zero, we have: $\lim_{\tau \to 0} f_{\tau, \rho_{N-k}}(\mathbf{x}) = \sigma \left( \frac{x_{\rho_{N-k}}}{\tau} + t(x_1, \ldots, x_N) \right) = 0, \quad \forall i \in [1, N-k]$ and $\lim_{\tau \to 0} f_{\tau, \rh

Figures (3)

  • Figure 1: Illustration of the SMART pruner. In the forward pass, the original weights (top left) are element-wise multiplied by a rescaled importance mask, $f_\tau(m)$, generated from a differentiable top-$k$ operator, to obtain the masked weights matrix (top right). In the backward pass, the original weights parameter $w$ and mask parameter $m$ are updated via back-propagation.
  • Figure 2: Training Flow of SMART Pruner. This illustrates a three-stage process: pretraining to develop the initial model, structural searching to determine the pruning structure, and fine-tuning to enhance performance post-pruning.
  • Figure 3: Illustration of the impact of the temperature parameter, $\tau$, on the SMART pruner. As $\tau$ decreases, the rescaled mask parameter, derived from the differentiable Top $k$ operator, more closely approximates binary values (0 or 1), promoting sparsity in the masked weights matrix.

Theorems & Definitions (12)

  • Theorem 3.1
  • Proposition 3.2
  • Theorem 3.3
  • Theorem 3.4
  • Theorem 3.5
  • Proposition 3.6
  • Theorem 3.7
  • proof
  • proof
  • proof
  • ...and 2 more