Performance Characterization of Containerized DNN Training and Inference on Edge Accelerators

TL;DR

The paper investigates whether Docker containerization introduces overheads for DNN training and inference on edge accelerators. Using a NVIDIA Jetson AGX Orin, it compares containerized and bare-metal executions across LeNet, MobileNet, and ResNet variants with MNIST, GLD23k, and ImageNet datasets, measuring time, resource usage, energy, and memory across multiple power modes. The key finding is that container overheads are negligible for both training and inference, with minibatch times, energy consumption, and memory footprints closely matching bare-metal runs. This supports container-based deployment for privacy-preserving, multi-tenant edge deployments and motivates future work on concurrent workloads and microVM-based isolation.

Abstract

Edge devices have typically been used for DNN inferencing. The increase in the compute power of accelerated edges is leading to their use in DNN training also. As privacy becomes a concern on multi-tenant edge devices, Docker containers provide a lightweight virtualization mechanism to sandbox models. But their overheads for edge devices are not yet explored. In this work, we study the impact of containerized DNN inference and training workloads on an NVIDIA AGX Orin edge device and contrast it against bare metal execution on running time, CPU, GPU and memory utilization, and energy consumption. Our analysis shows that there are negligible containerization overheads for individually running DNN training and inference workloads.

Figures (6)

• Figure 1: Avg. minibatch time & total energy for DNN training
• Figure 2: CPU, GPU utilization and power for DNN training
• Figure 3: Avg. minibatch time & total energy for DNN inf.
• Figure 4: CPU, GPU utilization and power for DNN inference
• Figure 5: Memory footprint for DNN training and inference