DNNShifter: An Efficient DNN Pruning System for Edge Computing

TL;DR

DNNShifter tackles edge deployment of production-grade DNNs by delivering an end-to-end pipeline that first generates a portfolio of sparse models through unstructured pruning integrated into training, then converts them into pruned, spatially efficient variants via structured pruning, and finally enables rapid on-device switching among variants through in-memory deflate/inflate techniques. This approach achieves substantial gains over NAS- and standard pruning-based baselines in portfolio generation speed, model size, and inference latency, while maintaining near-original accuracy. The framework demonstrates up to 93-fold faster portfolio generation than NAS, up to 5.14-fold smaller pruned models with 1.67x CPU and 1.32x GPU speedups, and up to 11.9x lower model-switching overhead with reduced memory usage. These results enable practical, adaptive edge AI workloads that can respond to changing resource and performance conditions with minimal downtime and resource footprint.

Abstract

Deep neural networks (DNNs) underpin many machine learning applications. Production quality DNN models achieve high inference accuracy by training millions of DNN parameters which has a significant resource footprint. This presents a challenge for resources operating at the extreme edge of the network, such as mobile and embedded devices that have limited computational and memory resources. To address this, models are pruned to create lightweight, more suitable variants for these devices. Existing pruning methods are unable to provide similar quality models compared to their unpruned counterparts without significant time costs and overheads or are limited to offline use cases. Our work rapidly derives suitable model variants while maintaining the accuracy of the original model. The model variants can be swapped quickly when system and network conditions change to match workload demand. This paper presents DNNShifter, an end-to-end DNN training, spatial pruning, and model switching system that addresses the challenges mentioned above. At the heart of DNNShifter is a novel methodology that prunes sparse models using structured pruning. The pruned model variants generated by DNNShifter are smaller in size and thus faster than dense and sparse model predecessors, making them suitable for inference at the edge while retaining near similar accuracy as of the original dense model. DNNShifter generates a portfolio of model variants that can be swiftly interchanged depending on operational conditions. DNNShifter produces pruned model variants up to 93x faster than conventional training methods. Compared to sparse models, the pruned model variants are up to 5.14x smaller and have a 1.67x inference latency speedup, with no compromise to sparse model accuracy. In addition, DNNShifter has up to 11.9x lower overhead for switching models and up to 3.8x lower memory utilisation than existing approaches.

Figures (7)

• Figure 1: Obtaining sparse and pruned models from pruning a dense model.
• Figure 2: Structured pruning zero-valued data structures obtained from unstructured pruning.
• Figure 3: Overview of the DNNShifter framework.
• Figure 4: The unstructured pruning method incorporated in DNNShifter uses the combined approach of repetitive training and model ranking between training iterations.
• Figure 5: Accuracy of unstructured pruning in DNNShifter against other methods as compression ratio increases; dashed line is baseline model accuracy.