Comprehensive Study on Performance Evaluation and Optimization of Model Compression: Bridging Traditional Deep Learning and Large Language Models

TL;DR

This work tackles the challenge of deploying modern deep learning models on edge devices by evaluating comprehensive compression strategies, focusing on post-training quantization and pruning across computer vision, natural language processing, and generative modeling. It contrasts Dynamic Range, Float16, and Int8 quantization alongside unstructured and structured pruning, probing effects on model size, accuracy, and inference latency, and extends the study to large language models with LoRA and low-rank adaptations. Key findings show that $4$x to $2$x size reductions are achievable via quantization with varying accuracy tradeoffs, while pruning can maintain or improve performance up to a threshold of sparsity but may not always translate to latency gains, especially on general-purpose CPUs. The paper provides practical guidance for edge deployment, highlighting hardware-specific considerations, tool support, and the potential of hybrid workflows (quantization plus pruning plus distillation) to deliver efficient, scalable LLMs and CNNs for real-world applications.

Abstract

Deep learning models have achieved tremendous success in most of the industries in recent years. The evolution of these models has also led to an increase in the model size and energy requirement, making it difficult to deploy in production on low compute devices. An increase in the number of connected devices around the world warrants compressed models that can be easily deployed at the local devices with low compute capacity and power accessibility. A wide range of solutions have been proposed by different researchers to reduce the size and complexity of such models, prominent among them are, Weight Quantization, Parameter Pruning, Network Pruning, low-rank representation, weights sharing, neural architecture search, knowledge distillation etc. In this research work, we investigate the performance impacts on various trained deep learning models, compressed using quantization and pruning techniques. We implemented both, quantization and pruning, compression techniques on popular deep learning models used in the image classification, object detection, language models and generative models-based problem statements. We also explored performance of various large language models (LLMs) after quantization and low rank adaptation. We used the standard evaluation metrics (model's size, accuracy, and inference time) for all the related problem statements and concluded this paper by discussing the challenges and future work.

Figures (19)

• Figure 1: Model format supported for TFLite converter
• Figure 2: Quantization Method Comparison
• Figure 3: Weight Distribution in a Conv. Layer Before Pruning
• Figure 4: Weight Distribution in a Conv. Layer After Pruning
• Figure 5: Metrics Monitored During Experiments