Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CompressNAS : A Fast and Efficient Technique for Model Compression using Decomposition

Sudhakar Sah, Nikhil Chabbra, Matthieu Durnerin

TL;DR

CompressNAS reframes Tucker-decomposition based CNN compression as a global neural architecture search problem, enabling globally consistent rank selection under memory/accuracy budgets. It integrates a MicroNAS-inspired workflow with a lightweight MSE-based accuracy proxy, fast flash estimation, and an ILP-driven search to rapidly generate compression configurations that preserve accuracy while achieving substantial parameter reductions. The approach yields competitive or superior results on ImageNet and COCO compared to prior Tucker-based methods, and introduces STResNet as a family of ultra-compact backbones well-suited for MCUs/NPUs and quantization-friendly deployment. Overall, CompressNAS provides a practical, single-search framework that supports budget-aware, fine-grain rank optimization and scalable model generation across multiple compression targets.

Abstract

Deep Convolutional Neural Networks (CNNs) are increasingly difficult to deploy on microcontrollers (MCUs) and lightweight NPUs (Neural Processing Units) due to their growing size and compute demands. Low-rank tensor decomposition, such as Tucker factorization, is a promising way to reduce parameters and operations with reasonable accuracy loss. However, existing approaches select ranks locally and often ignore global trade-offs between compression and accuracy. We introduce CompressNAS, a MicroNAS-inspired framework that treats rank selection as a global search problem. CompressNAS employs a fast accuracy estimator to evaluate candidate decompositions, enabling efficient yet exhaustive rank exploration under memory and accuracy constraints. In ImageNet, CompressNAS compresses ResNet-18 by 8x with less than 4% accuracy drop; on COCO, we achieve 2x compression of YOLOv5s without any accuracy drop and 2x compression of YOLOv5n with a 2.5% drop. Finally, we present a new family of compressed models, STResNet, with competitive performance compared to other efficient models.

CompressNAS : A Fast and Efficient Technique for Model Compression using Decomposition

TL;DR

CompressNAS reframes Tucker-decomposition based CNN compression as a global neural architecture search problem, enabling globally consistent rank selection under memory/accuracy budgets. It integrates a MicroNAS-inspired workflow with a lightweight MSE-based accuracy proxy, fast flash estimation, and an ILP-driven search to rapidly generate compression configurations that preserve accuracy while achieving substantial parameter reductions. The approach yields competitive or superior results on ImageNet and COCO compared to prior Tucker-based methods, and introduces STResNet as a family of ultra-compact backbones well-suited for MCUs/NPUs and quantization-friendly deployment. Overall, CompressNAS provides a practical, single-search framework that supports budget-aware, fine-grain rank optimization and scalable model generation across multiple compression targets.

Abstract

Deep Convolutional Neural Networks (CNNs) are increasingly difficult to deploy on microcontrollers (MCUs) and lightweight NPUs (Neural Processing Units) due to their growing size and compute demands. Low-rank tensor decomposition, such as Tucker factorization, is a promising way to reduce parameters and operations with reasonable accuracy loss. However, existing approaches select ranks locally and often ignore global trade-offs between compression and accuracy. We introduce CompressNAS, a MicroNAS-inspired framework that treats rank selection as a global search problem. CompressNAS employs a fast accuracy estimator to evaluate candidate decompositions, enabling efficient yet exhaustive rank exploration under memory and accuracy constraints. In ImageNet, CompressNAS compresses ResNet-18 by 8x with less than 4% accuracy drop; on COCO, we achieve 2x compression of YOLOv5s without any accuracy drop and 2x compression of YOLOv5n with a 2.5% drop. Finally, we present a new family of compressed models, STResNet, with competitive performance compared to other efficient models.

Paper Structure

This paper contains 22 sections, 4 equations, 6 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Tensor Decomposition for CNN Compression.
  4. Automatic and Budget-Aware Rank Selection.
  5. Zero-Cost Proxies and NAS-Inspired Rank Search.
  6. Efficient models.
  7. Positioning.
  8. Tucker Decomposition
  9. CompressNAS Architecture
  10. Network Proposals
  11. Accuracy Estimator
  12. Flash Estimator
  13. Neural Architecture Search
  14. Optimized Model Architecture
  15. Results
  16. ...and 7 more sections

Figures (6)

  • Figure 1: Decomposition of a CNN layer : $R1$ and $R2$ are selected ranks for decomposed layers
  • Figure 2: CompressNAS : Model Proposal Generation and Profiling
  • Figure 3: Comparison of different zero cost estimator scores for 7 initial layers of YOLOv5n
  • Figure 4: Mean Square Error based Accuracy Estimator
  • Figure 5: CompressNAS : Optimized Model Creation
  • ...and 1 more figures