Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Systematic Characterization of Minimal Deep Learning Architectures: A Unified Analysis of Convergence, Pruning, and Quantization

Ziwei Zheng, Huizhi Liang, Vaclav Snasel, Vito Latora, Panos Pardalos, Giuseppe Nicosia, Varun Ojha

TL;DR

The paper tackles the challenge of identifying minimal yet reliable neural architectures for image classification by introducing a unified framework that jointly analyzes convergence, pruning, and quantization across DNNs, CNNs, and ViTs. A two-phase design sweep, plus a parameter-budget study, reveals a two-stage learning dynamic with three regimes (unstable, stable learning, overfitting) and shows that beyond a critical parameter count, architectural topology has limited impact on convergence. It empirically characterizes three regime-specific pruning tolerances and quantization robustness, finding that deeper models tolerate >60% pruning while 8-bit quantization remains more benign on simpler tasks but degrades harder datasets like CIFAR-10, especially for ViTs at small widths. The integrated methodology provides actionable guidance for constructing compact, stable models under pruning and low-precision constraints and offers a workflow for designing minimal yet robust architectures across DNNs, CNNs, and ViTs.

Abstract

Deep learning networks excel at classification, yet identifying minimal architectures that reliably solve a task remains challenging. We present a computational methodology for systematically exploring and analyzing the relationships among convergence, pruning, and quantization. The workflow first performs a structured design sweep across a large set of architectures, then evaluates convergence behavior, pruning sensitivity, and quantization robustness on representative models. Focusing on well-known image classification of increasing complexity, and across Deep Neural Networks, Convolutional Neural Networks, and Vision Transformers, our initial results show that, despite architectural diversity, performance is largely invariant and learning dynamics consistently exhibit three regimes: unstable, learning, and overfitting. We further characterize the minimal learnable parameters required for stable learning, uncover distinct convergence and pruning phases, and quantify the effect of reduced numeric precision on trainable parameters. Aligning with intuition, the results confirm that deeper architectures are more resilient to pruning than shallower ones, with parameter redundancy as high as 60%, and quantization impacts models with fewer learnable parameters more severely and has a larger effect on harder image datasets. These findings provide actionable guidance for selecting compact, stable models under pruning and low-precision constraints in image classification.

Systematic Characterization of Minimal Deep Learning Architectures: A Unified Analysis of Convergence, Pruning, and Quantization

TL;DR

The paper tackles the challenge of identifying minimal yet reliable neural architectures for image classification by introducing a unified framework that jointly analyzes convergence, pruning, and quantization across DNNs, CNNs, and ViTs. A two-phase design sweep, plus a parameter-budget study, reveals a two-stage learning dynamic with three regimes (unstable, stable learning, overfitting) and shows that beyond a critical parameter count, architectural topology has limited impact on convergence. It empirically characterizes three regime-specific pruning tolerances and quantization robustness, finding that deeper models tolerate >60% pruning while 8-bit quantization remains more benign on simpler tasks but degrades harder datasets like CIFAR-10, especially for ViTs at small widths. The integrated methodology provides actionable guidance for constructing compact, stable models under pruning and low-precision constraints and offers a workflow for designing minimal yet robust architectures across DNNs, CNNs, and ViTs.

Abstract

Deep learning networks excel at classification, yet identifying minimal architectures that reliably solve a task remains challenging. We present a computational methodology for systematically exploring and analyzing the relationships among convergence, pruning, and quantization. The workflow first performs a structured design sweep across a large set of architectures, then evaluates convergence behavior, pruning sensitivity, and quantization robustness on representative models. Focusing on well-known image classification of increasing complexity, and across Deep Neural Networks, Convolutional Neural Networks, and Vision Transformers, our initial results show that, despite architectural diversity, performance is largely invariant and learning dynamics consistently exhibit three regimes: unstable, learning, and overfitting. We further characterize the minimal learnable parameters required for stable learning, uncover distinct convergence and pruning phases, and quantify the effect of reduced numeric precision on trainable parameters. Aligning with intuition, the results confirm that deeper architectures are more resilient to pruning than shallower ones, with parameter redundancy as high as 60%, and quantization impacts models with fewer learnable parameters more severely and has a larger effect on harder image datasets. These findings provide actionable guidance for selecting compact, stable models under pruning and low-precision constraints in image classification.
Paper Structure (19 sections, 5 figures, 1 table)

This paper contains 19 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Systematic Characterization
  5. Convergence phases for stable learning
  6. Pruning phase of stable learning
  7. Quantization phase stable learning
  8. CNN and ViT architectures
  9. CNN network
  10. ViT network
  11. Experimental Setup
  12. DNN-specific configuration
  13. CNN-specific configuration.
  14. ViT-specific configuration.
  15. Results and Analysis
  16. ...and 4 more sections

Figures (5)

  • Figure 1: 1-layer whole network, pruned network, and quantized network (see Sec. \ref{['sec:method']} for details).
  • Figure 2: Network architecture design and performance. (a) The number indicates the number of hidden layers. The weight matrix shapes $\vartriangleleft$, $\vartriangleright$, and $\square$ stand for an increase, decrease, and an equal number of nodes in the consecutive layers, and $\mid$ represents a single hidden layer. (b) Left to right: convergence performance of 1-layer to 4-layer networks. The vertical dotted lines separate different ranges of the total number of learnable parameters.
  • Figure 3: Convergence, Pruning, and Quantization Results on DNN. The solid lines indicate the average test accuracy, and the shaded area is $\pm$ standard deviation. Colors represent the network with hidden layers of $1,$$$2,$$$3,$ and $4,$ respectively. The curves reflect changes in network performance and stability as the number of nodes in the first hidden layer increases ($x$-axis in Convergence and Quantization). A value 5 on the $x-axis$ corresponds to architectures {$5$}, {$5\times5$}, {$5\times5\times5$}, and {$5\times5\times5\times5$} respectively to 1, 2, 3, and 4, layered DNNs, and the input and output layers varied respectively to datasets. For pruning, the $x-axis$ represents the pruning ratio, and dotted vertical lines indicate the pruning ratio that matches the curves for different layers when the same accuracy is achieved. Symbol $\theta$ indicates average learnable parameters.
  • Figure 4: Convergence, Pruning, and Quantization Results on CNN. The curves reflect network performance and stability changes as the number of channels in the first convolution layer increases ($x-axis$ in Convergence and Quantization).
  • Figure 5: Convergence, Pruning, and Quantization Results on VIT. The curves reflect network performance and stability changes as the number of embedding channels increases ($x-axis$ in Convergence and Quantization).