Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sensitivity-Aware Mixed-Precision Quantization and Width Optimization of Deep Neural Networks Through Cluster-Based Tree-Structured Parzen Estimation

Seyedarmin Azizi, Mahdi Nazemi, Arash Fayyazi, Massoud Pedram

TL;DR

This work tackles the problem of efficiently compressing and accelerating DNNs by jointly optimizing per-layer bit-width and layer-width under hardware constraints. It combines Hessian-based search-space pruning with a cluster-based dual-threshold Parzen estimator (K-Means TPE) to navigate a drastically reduced design space while considering real hardware performance. The proposed hardware-aware objective guides the search, yielding substantial gains in model size and latency with preserved accuracy across ImageNet and CIFAR benchmarks; the method achieves a 12× average reduction in search time and a 20% model-size reduction relative to state-of-the-art compression techniques. These contributions facilitate rapid, resource-efficient neural network design and deployment on edge and embedded platforms.

Abstract

As the complexity and computational demands of deep learning models rise, the need for effective optimization methods for neural network designs becomes paramount. This work introduces an innovative search mechanism for automatically selecting the best bit-width and layer-width for individual neural network layers. This leads to a marked enhancement in deep neural network efficiency. The search domain is strategically reduced by leveraging Hessian-based pruning, ensuring the removal of non-crucial parameters. Subsequently, we detail the development of surrogate models for favorable and unfavorable outcomes by employing a cluster-based tree-structured Parzen estimator. This strategy allows for a streamlined exploration of architectural possibilities and swift pinpointing of top-performing designs. Through rigorous testing on well-known datasets, our method proves its distinct advantage over existing methods. Compared to leading compression strategies, our approach records an impressive 20% decrease in model size without compromising accuracy. Additionally, our method boasts a 12x reduction in search time relative to the best search-focused strategies currently available. As a result, our proposed method represents a leap forward in neural network design optimization, paving the way for quick model design and implementation in settings with limited resources, thereby propelling the potential of scalable deep learning solutions.

Sensitivity-Aware Mixed-Precision Quantization and Width Optimization of Deep Neural Networks Through Cluster-Based Tree-Structured Parzen Estimation

TL;DR

This work tackles the problem of efficiently compressing and accelerating DNNs by jointly optimizing per-layer bit-width and layer-width under hardware constraints. It combines Hessian-based search-space pruning with a cluster-based dual-threshold Parzen estimator (K-Means TPE) to navigate a drastically reduced design space while considering real hardware performance. The proposed hardware-aware objective guides the search, yielding substantial gains in model size and latency with preserved accuracy across ImageNet and CIFAR benchmarks; the method achieves a 12× average reduction in search time and a 20% model-size reduction relative to state-of-the-art compression techniques. These contributions facilitate rapid, resource-efficient neural network design and deployment on edge and embedded platforms.

Abstract

As the complexity and computational demands of deep learning models rise, the need for effective optimization methods for neural network designs becomes paramount. This work introduces an innovative search mechanism for automatically selecting the best bit-width and layer-width for individual neural network layers. This leads to a marked enhancement in deep neural network efficiency. The search domain is strategically reduced by leveraging Hessian-based pruning, ensuring the removal of non-crucial parameters. Subsequently, we detail the development of surrogate models for favorable and unfavorable outcomes by employing a cluster-based tree-structured Parzen estimator. This strategy allows for a streamlined exploration of architectural possibilities and swift pinpointing of top-performing designs. Through rigorous testing on well-known datasets, our method proves its distinct advantage over existing methods. Compared to leading compression strategies, our approach records an impressive 20% decrease in model size without compromising accuracy. Additionally, our method boasts a 12x reduction in search time relative to the best search-focused strategies currently available. As a result, our proposed method represents a leap forward in neural network design optimization, paving the way for quick model design and implementation in settings with limited resources, thereby propelling the potential of scalable deep learning solutions.
Paper Structure (13 sections, 1 theorem, 7 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 13 sections, 1 theorem, 7 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Method
  4. Hessian-Based Search Space Pruning
  5. $K$-Means TPE
  6. Hardware-Aware Objective Function
  7. Results & Discussion
  8. Convergence of $K$-Means TPE
  9. Mixed-Precision Quantization and Layer-Width Scaling
  10. ImageNet
  11. CIFAR-100
  12. CIFAR-10
  13. Conclusions

Key Result

Lemma 1

The maximum error induced in a DNN's output by unit perturbation in a layer's parameters is bounded by the trace value of the Hessian matrix of the loss with respect to that layer's parameters.

Figures (4)

  • Figure 1: Distribution of weights in three representative layers of the MobileNetV1 architecture trained on the CIFAR-100 dataset.
  • Figure 2: Four-bit operand and operation packing. The design yields the computations required for two rows of convolutional kernels every two cycles.
  • Figure 3: Comparison of the convergence speed of TPE and $k$-means TPE for different machine learning algorithms and on Iris, Titanic, and CIFAR-100 datasets.
  • Figure 4: The space for ResNet-18 compression and the output model

Theorems & Definitions (2)

  • Lemma 1
  • proof