Decay Pruning Method: Smooth Pruning With a Self-Rectifying Procedure

TL;DR

The Decay Pruning Method (DPM), a novel smooth pruning approach with a self-rectifying mechanism that demonstrates strong generalizability and can be seamlessly integrated with various existing pruning frameworks is introduced.

Abstract

Current structured pruning methods often result in considerable accuracy drops due to abrupt network changes and loss of information from pruned structures. To address these issues, we introduce the Decay Pruning Method (DPM), a novel smooth pruning approach with a self-rectifying mechanism. DPM consists of two key components: (i) Smooth Pruning: It converts conventional single-step pruning into multi-step smooth pruning, gradually reducing redundant structures to zero over N steps with ongoing optimization. (ii) Self-Rectifying: This procedure further enhances the aforementioned process by rectifying sub-optimal pruning based on gradient information. Our approach demonstrates strong generalizability and can be easily integrated with various existing pruning methods. We validate the effectiveness of DPM by integrating it with three popular pruning methods: OTOv2, Depgraph, and Gate Decorator. Experimental results show consistent improvements in performance compared to the original pruning methods, along with further reductions of FLOPs in most scenarios.

Figures (6)

• Figure 1: Single-step pruning vs. Decay pruning method. (a) Once pruning targets are identified, the single-step pruning process removes them in one action, risking abrupt network changes and irreversible information loss. (b) In contrast, the Decay Pruning Method gradually reduces the weight values of pruning structures over $N$ steps, employing a gradient-driven self-rectifying procedure to identify and rectify sub-optimal decisions.
• Figure 2: An illustration of the smooth pruning procedure (SP) with $N=3$.
• Figure 3: Illustrations of Actual escaping length.
• Figure 4: Illustrations of the impact of varying decaying step $N$ from 3 to 128. Red lines indicate the accuracy, and blue lines represent the FLOPs of the pruned networks.
• Figure 5: Illustrations of the impact of varying $T_{rate}$ and $T_{len}$. Red lines indicate the accuracy, and blue lines represent the FLOPs of the pruned networks. The total released amount is depicted with histograms.