CAMP-HiVe: Cyclic Pair Merging based Efficient DNN Pruning with Hessian-Vector Approximation for Resource-Constrained Systems

TL;DR

This work tackles the challenge of deploying large deep neural networks on resource-constrained devices by introducing CAMP-HiVe, a pruning method that uses Hessian-vector products to capture curvature information and a cyclic pairing scheme to merge less significant weights into more significant ones. A power-iteration approach estimates the dominant curvature direction without forming the full Hessian, guiding weight importance via per-weight scores and a percentile threshold. The method achieves substantial reductions in FLOPs and latency while preserving or improving accuracy across ResNet-18/56 and MobileNetV2 on CIFAR-10, CIFAR-100, and ImageNet, with demonstrated gains on edge devices. By combining second-order sensitivity with a dynamic redistribution strategy, CAMP-HiVe outperforms state-of-the-art pruning methods and offers practical benefits for real-world, hardware-constrained deployments.

Abstract

Deep learning algorithms are becoming an essential component of many artificial intelligence (AI) driven applications, many of which run on resource-constrained and energy-constrained systems. For efficient deployment of these algorithms, although different techniques for the compression of neural network models are proposed, neural pruning is one of the fastest and effective methods, which can provide a high compression gain with minimal cost. To harness enhanced performance gain with respect to model complexity, we propose a novel neural network pruning approach utilizing Hessian-vector products that approximate crucial curvature information in the loss function, which significantly reduces the computation demands. By employing a power iteration method, our algorithm effectively identifies and preserves the essential information, ensuring a balanced trade-off between model accuracy and computational efficiency. Herein, we introduce CAMP-HiVe, a cyclic pair merging-based pruning with Hessian Vector approximation by iteratively consolidating weight pairs, combining significant and less significant weights, thus effectively streamlining the model while preserving its performance. This dynamic, adaptive framework allows for real-time adjustment of weight significance, ensuring that only the most critical parameters are retained. Our experimental results demonstrate that our proposed method achieves significant reductions in computational requirements while maintaining high performance across different neural network architectures, e.g., ResNet18, ResNet56, and MobileNetv2, on standard benchmark datasets, e.g., CIFAR10, CIFAR-100, and ImageNet, and it outperforms the existing state-of-the-art neural pruning methods.

Figures (6)

• Figure 1: Accuracy vs. FLOPs Reduction Comparision between different models (Bubble $\propto |\Delta\mathrm{-Acc}|$). More detailed results have been presented in Tables \ref{['Cifat10_100_DataResult']} and \ref{['imagenetDataResult']}.
• Figure 2: MAD across different pruning levels on MobileNetV2
• Figure 3: MAD across different pruning levels on Resnet-56
• Figure 4: Inference vs pruning percentage comparison across different stages of pruned models on MobileNet-V2 for various edge devices( (a) Nvidia Jetson AGX-Orin 16GB, (b) Nvidia Jetson AGX-Orin-32GB, (c) Nvidia Jetson Orin-Nano-4GB).
• Figure 5: Power vs Inference comparison across different stages of pruned models on MobileNet-V2 for various edge devices: (a) Nvidia Jetson AGX-Orin 16GB, (b) Nvidia Jetson AGX-Orin-32GB, (c) Nvidia Jetson Orin-Nano-4GB.