Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Achieving Pareto Optimality using Efficient Parameter Reduction for DNNs in Resource-Constrained Edge Environment

Atah Nuh Mih, Alireza Rahimi, Asfia Kawnine, Francis Palma, Monica Wachowicz, Rickey Dubay, Hung Cao

TL;DR

The paper tackles enabling on-device training of DNNs on resource-constrained edge devices by optimizing an existing backbone, Xception, through horizontal compact network design inspired by SqueezeNet. It reduces parameters by replacing 3x3 convolutions with 1x1 filters and by trimming channel counts while preserving the macro-architecture, achieving higher or comparable accuracy with lower memory and competitive latency on Caltech-101 and PCB defect detection. The results demonstrate Pareto-optimality in accuracy-memory trade-offs and highlight the nuanced effects of transfer learning on memory and accuracy. The work suggests future automation via neural architecture search and vertical compact designs to further enhance edge-training efficiency and performance.

Abstract

This paper proposes an optimization of an existing Deep Neural Network (DNN) that improves its hardware utilization and facilitates on-device training for resource-constrained edge environments. We implement efficient parameter reduction strategies on Xception that shrink the model size without sacrificing accuracy, thus decreasing memory utilization during training. We evaluate our model in two experiments: Caltech-101 image classification and PCB defect detection and compare its performance against the original Xception and lightweight models, EfficientNetV2B1 and MobileNetV2. The results of the Caltech-101 image classification show that our model has a better test accuracy (76.21%) than Xception (75.89%), uses less memory on average (847.9MB) than Xception (874.6MB), and has faster training and inference times. The lightweight models overfit with EfficientNetV2B1 having a 30.52% test accuracy and MobileNetV2 having a 58.11% test accuracy. Both lightweight models have better memory usage than our model and Xception. On the PCB defect detection, our model has the best test accuracy (90.30%), compared to Xception (88.10%), EfficientNetV2B1 (55.25%), and MobileNetV2 (50.50%). MobileNetV2 has the least average memory usage (849.4MB), followed by our model (865.8MB), then EfficientNetV2B1 (874.8MB), and Xception has the highest (893.6MB). We further experiment with pre-trained weights and observe that memory usage decreases thereby showing the benefits of transfer learning. A Pareto analysis of the models' performance shows that our optimized model architecture satisfies accuracy and low memory utilization objectives.

Achieving Pareto Optimality using Efficient Parameter Reduction for DNNs in Resource-Constrained Edge Environment

TL;DR

The paper tackles enabling on-device training of DNNs on resource-constrained edge devices by optimizing an existing backbone, Xception, through horizontal compact network design inspired by SqueezeNet. It reduces parameters by replacing 3x3 convolutions with 1x1 filters and by trimming channel counts while preserving the macro-architecture, achieving higher or comparable accuracy with lower memory and competitive latency on Caltech-101 and PCB defect detection. The results demonstrate Pareto-optimality in accuracy-memory trade-offs and highlight the nuanced effects of transfer learning on memory and accuracy. The work suggests future automation via neural architecture search and vertical compact designs to further enhance edge-training efficiency and performance.

Abstract

This paper proposes an optimization of an existing Deep Neural Network (DNN) that improves its hardware utilization and facilitates on-device training for resource-constrained edge environments. We implement efficient parameter reduction strategies on Xception that shrink the model size without sacrificing accuracy, thus decreasing memory utilization during training. We evaluate our model in two experiments: Caltech-101 image classification and PCB defect detection and compare its performance against the original Xception and lightweight models, EfficientNetV2B1 and MobileNetV2. The results of the Caltech-101 image classification show that our model has a better test accuracy (76.21%) than Xception (75.89%), uses less memory on average (847.9MB) than Xception (874.6MB), and has faster training and inference times. The lightweight models overfit with EfficientNetV2B1 having a 30.52% test accuracy and MobileNetV2 having a 58.11% test accuracy. Both lightweight models have better memory usage than our model and Xception. On the PCB defect detection, our model has the best test accuracy (90.30%), compared to Xception (88.10%), EfficientNetV2B1 (55.25%), and MobileNetV2 (50.50%). MobileNetV2 has the least average memory usage (849.4MB), followed by our model (865.8MB), then EfficientNetV2B1 (874.8MB), and Xception has the highest (893.6MB). We further experiment with pre-trained weights and observe that memory usage decreases thereby showing the benefits of transfer learning. A Pareto analysis of the models' performance shows that our optimized model architecture satisfies accuracy and low memory utilization objectives.
Paper Structure (18 sections, 2 equations, 4 figures, 2 tables)

This paper contains 18 sections, 2 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Model Optimization using Post-Training Quantization
  4. Model Optimization with Neural Architecture Search
  5. AI Models at the Edge
  6. Proposed Approach and Implementation
  7. Xception
  8. SqueezeNet
  9. Optimizing Xception
  10. Experiments and Results
  11. Experiment 1: Image Classification with Caltech-101
  12. Experiment 2: PCB Defect Detection
  13. Non Pre-trained Models
  14. Pre-trained Models
  15. Discussion and Conclusion
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Optimizing a model using Compact Network Design.
  • Figure 2: Implementation of an optimized DNN.
  • Figure 3: Accuracy and Memory Usage Patterns During Training (compared with original Xception architecture as baseline and other lightweight models, EfficientNetV2B1 and MobileNetV2)
  • Figure 4: Dual-objective Analysis of Models for Accuracy and Memory Usage. In both comparisons, we define our accuracy frontier at 70% and our memory usage frontier at the midpoint of the min and max values.