Multi-Dimensional Pruning: Joint Channel, Layer and Block Pruning with Latency Constraint

TL;DR

This work introduces Multi-Dimensional Pruning (MDP), a latency-aware framework that jointly prunes channels, layers, and blocks under a latency budget using a single-pass Mixed-Integer Nonlinear Program. By grouping layers into blocks and employing bilayer configuration latency through a latency matrix $\mathbf{C}_l$, MDP achieves globally optimal pruned structures via Outer Approximation, followed by extraction and fine-tuning. Across ImageNet, Pascal VOC, and Nuscenes, MDP delivers state-of-the-art accuracy-latency trade-offs at high pruning ratios, outperforming prior hardware-aware methods like HALP and showing substantial speedups on both CNN and hybrid CNN-Transformer models. The approach offers practical impact for real-time edge deployment, providing a robust, hardware-aware pruning strategy that can adapt to different platforms with a single optimization pass.

Abstract

As we push the boundaries of performance in various vision tasks, the models grow in size correspondingly. To keep up with this growth, we need very aggressive pruning techniques for efficient inference and deployment on edge devices. Existing pruning approaches are limited to channel pruning and struggle with aggressive parameter reductions. In this paper, we propose a novel multi-dimensional pruning framework that jointly optimizes pruning across channels, layers, and blocks while adhering to latency constraints. We develop a latency modeling technique that accurately captures model-wide latency variations during pruning, which is crucial for achieving an optimal latency-accuracy trade-offs at high pruning ratio. We reformulate pruning as a Mixed-Integer Nonlinear Program (MINLP) to efficiently determine the optimal pruned structure with only a single pass. Our extensive results demonstrate substantial improvements over previous methods, particularly at large pruning ratios. In classification, our method significantly outperforms prior art HALP with a Top-1 accuracy of 70.0(v.s. 68.6) and an FPS of 5262 im/s(v.s. 4101 im/s). In 3D object detection, we establish a new state-of-the-art by pruning StreamPETR at a 45% pruning ratio, achieving higher FPS (37.3 vs. 31.7) and mAP (0.451 vs. 0.449) than the dense baseline.

Figures (6)

• Figure 1: MDP exhibits Pareto dominance across different tasks. In contrast to existing methods: [Left] On Imagenet classification, we achieve a 6.2% relative accuracy gain with a 2.6% FPS speedup, and even greater gains at higher pruning ratio: a 2% relative gain with a substantial 28.3% FPS speedup. [Right] On NuScenes 3D object detection, we observe a 5.6% relative mAP improvement alongside a 1.8% FPS increase.
• Figure 2: Paradigm of our proposed method MDP. We start by computing layer importance and constructing latency cost matrices for each layer. We then group layers within the same block and solve an MINLP to optimize pruning decisions at both channel and block levels. Finally, we extract the pruned subnetwork and finetune it.
• Figure 3: PascalVOC results with SSD512. FPS is measured on NVIDIA TITAN V with batch size of $256$. Ours achieve much better mAP-FPS and mAP-FLOPs tradeoffs than the baselines.
• Figure 4: Ablation study results on ImageNet with ResNet50. We show results of each improvement acting individually. Top-right is better.
• Figure 5: Pruned architecture.