Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hidden costs for inference with deep network on embedded system devices

Chankyu Lee, Woohyun Choi, Sangwook Park

TL;DR

This paper investigates why Multiply-Accumulate (MAC) counts fail to predict inference time for deep networks on embedded devices. By profiling 10 CNN architectures on CIFAR-100 using an embedded platform, it separates MAC-based complexity from actual latency and reveals that tensor operations between tensors and memory management dominate in many cases. The study shows that CNNs with lower MACs can still exhibit long inference times due to overheads like slicing, memory allocation, and non-convolution operations, highlighting the need for more holistic performance metrics. The findings have practical implications for deploying real-time deep learning on resource-constrained hardware, guiding model design and deployment strategies beyond MAC optimization.

Abstract

This study evaluates the inference performance of various deep learning models under an embedded system environment. In previous works, Multiply-Accumulate operation is typically used to measure computational load of a deep model. According to this study, however, this metric has a limitation to estimate inference time on embedded devices. This paper poses the question of what aspects are overlooked when expressed in terms of Multiply-Accumulate operations. In experiments, an image classification task is performed on an embedded system device using the CIFAR-100 dataset to compare and analyze the inference times of ten deep models with the theoretically calculated Multiply-Accumulate operations for each model. The results highlight the importance of considering additional computations between tensors when optimizing deep learning models for real-time performing in embedded systems.

Hidden costs for inference with deep network on embedded system devices

TL;DR

This paper investigates why Multiply-Accumulate (MAC) counts fail to predict inference time for deep networks on embedded devices. By profiling 10 CNN architectures on CIFAR-100 using an embedded platform, it separates MAC-based complexity from actual latency and reveals that tensor operations between tensors and memory management dominate in many cases. The study shows that CNNs with lower MACs can still exhibit long inference times due to overheads like slicing, memory allocation, and non-convolution operations, highlighting the need for more holistic performance metrics. The findings have practical implications for deploying real-time deep learning on resource-constrained hardware, guiding model design and deployment strategies beyond MAC optimization.

Abstract

This study evaluates the inference performance of various deep learning models under an embedded system environment. In previous works, Multiply-Accumulate operation is typically used to measure computational load of a deep model. According to this study, however, this metric has a limitation to estimate inference time on embedded devices. This paper poses the question of what aspects are overlooked when expressed in terms of Multiply-Accumulate operations. In experiments, an image classification task is performed on an embedded system device using the CIFAR-100 dataset to compare and analyze the inference times of ten deep models with the theoretically calculated Multiply-Accumulate operations for each model. The results highlight the importance of considering additional computations between tensors when optimizing deep learning models for real-time performing in embedded systems.
Paper Structure (7 sections, 2 equations, 3 figures, 3 tables)

This paper contains 7 sections, 2 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Materials and Methods
  3. Database & Hardware specification
  4. Models
  5. Survey of inference time and MAC
  6. Results
  7. Conclusion

Figures (3)

  • Figure 1: Experiment environment performed on embedded system
  • Figure 2: Distributions of model accuracy and computation speed for 10-deep models along with Pearson correlation coefficient: (a) Accuracy vs size, (b) Size vs Computations, (c) Accuracy vs time, (d) Time vs computations
  • Figure 3: The ratio of computation components during model inference.