Attentive Fine-Grained Structured Sparsity for Image Restoration

TL;DR

This work proposes a novel pruning method that determines the pruning ratio for N:M structured sparsity at each layer of an image restoration network and demonstrates the efficacy of this method which outperforms previous pruning methods significantly.

Abstract

Image restoration tasks have witnessed great performance improvement in recent years by developing large deep models. Despite the outstanding performance, the heavy computation demanded by the deep models has restricted the application of image restoration. To lift the restriction, it is required to reduce the size of the networks while maintaining accuracy. Recently, N:M structured pruning has appeared as one of the effective and practical pruning approaches for making the model efficient with the accuracy constraint. However, it fails to account for different computational complexities and performance requirements for different layers of an image restoration network. To further optimize the trade-off between the efficiency and the restoration accuracy, we propose a novel pruning method that determines the pruning ratio for N:M structured sparsity at each layer. Extensive experimental results on super-resolution and deblurring tasks demonstrate the efficacy of our method which outperforms previous pruning methods significantly. PyTorch implementation for the proposed methods is available at https://github.com/JungHunOh/SLS_CVPR2022.

Figures (8)

• Figure 1: Trade-off between image restoration performance (PSNR) vs computational costs (MACs) on super-resolution (top) and deblurring tasks (bottom). We compare our method to the magnitude-based filter pruning li2017pruning and the existing methods on $N$:$M$ sparsity (One-shot pruning mishra2021accelerating and SR-STE zhou2021learning).
• Figure 2: (a) Visual illustration for $N$:$M$ sparsity mishra2021accelerating, where $N=3$ and $M=4$. We illustrate the sparse multiplication process only for the first row of the weight, where the non-zero weights and the input features at the corresponding positions are highlighted in green boxes. $D$ refers to the spatial dimension of the input feature. (e.g., input height $\times$ input width in a convolutional layer.) (b) Overview of the proposed method, SLS, which decomposes the weights into $M$ ($M = 4$ in this case) groups based on the weight magnitude (Equation \ref{['eq:decompositon']}). According to Equation \ref{['eq:pruning_problem']}, the final pruned weights are constructed based on the binary mask $\textbf{b}$, each value of which controls whether the corresponding group will be used. The binary mask will be generated based on the priority of each importance group, following the operations outlined by Equation \ref{['eq:ste']} and \ref{['eq:markov']}. We illustrate the case when the searched sparsity is 3:4.
• Figure 3: Visualization of the proposed adaptive inference method. Given the 3 images on the right side of the figure and 4 model candidates (75%, 87.5%, 93.75% pruned CARN ahn2018fast and bicubic upsampler), the trained MSE estimators estimate MSE between the restored image and the ground truth. Then, our method selects one model by Equation \ref{['eq:adaptive_inference']}. By adjusting $\beta$, one can determine whether to focus on efficiency or accuracy.
• Figure 4: Qualitative comparisons with the existing pruning methods. The first two rows show image super-resolution results from RFDN with the scaling factor of 4. The last two rows show image deblurring results from DMPHN. For each task, all pruned models have almost the same computational costs (1/16 of the original value) with respect to GMACs.
• Figure 5: Analysis on the searched layer-wise $N$:$M$ sparsity. The results are obtained by training RFDN model with $C_{target}=\frac{1}{8}C_{original}$. We visualize the layer-wise pruning ratios in terms of (a) GMACs and (b) $N$. The layer is sorted with respect to MACs before pruning. The highlighted bar by the red arrow indicates the result of the last upsampling layer in RFDN. The brown line in (b) is for the comparison with the uniform pruning mishra2021acceleratingzhou2021learning.