Optimizing DNN Inference on Multi-Accelerator SoCs at Training-time

TL;DR

ODiMO tackles the challenge of mapping DNN inference onto multi-CU heterogeneous edge SoCs while accounting for accuracy. It formulates a training-time, gradient-based optimization that partitions each layer at fine granularity across CUs with incompatible data formats or specialized units, balancing latency or energy with accuracy. The approach yields rich Pareto fronts across CIFAR-10/100 and ImageNet on DIANA and Darkside, achieving up to $8\times$ latency reduction and up to $50.8\times$ energy savings with minimal accuracy loss. Practical validation includes hardware-model micro-benchmarks and deployment on DIANA, underscoring ODiMO’s potential for efficient edge inference on diverse heterogeneous platforms.

Abstract

The demand for executing Deep Neural Networks (DNNs) with low latency and minimal power consumption at the edge has led to the development of advanced heterogeneous Systems-on-Chips (SoCs) that incorporate multiple specialized computing units (CUs), such as accelerators. Offloading DNN computations to a specific CU from the available set often exposes accuracy vs efficiency trade-offs, due to differences in their supported operations (e.g., standard vs. depthwise convolution) or data representations (e.g., more/less aggressively quantized). A challenging yet unresolved issue is how to map a DNN onto these multi-CU systems to maximally exploit the parallelization possibilities while taking accuracy into account. To address this problem, we present ODiMO, a hardware-aware tool that efficiently explores fine-grain mapping of DNNs among various on-chip CUs, during the training phase. ODiMO strategically splits individual layers of the neural network and executes them in parallel on the multiple available CUs, aiming to balance the total inference energy consumption or latency with the resulting accuracy, impacted by the unique features of the different hardware units. We test our approach on CIFAR-10, CIFAR-100, and ImageNet, targeting two open-source heterogeneous SoCs, i.e., DIANA and Darkside. We obtain a rich collection of Pareto-optimal networks in the accuracy vs. energy or latency space. We show that ODiMO reduces the latency of a DNN executed on the Darkside SoC by up to 8x at iso-accuracy, compared to manual heuristic mappings. When targeting energy, on the same SoC, ODiMO produced up to 50.8x more efficient mappings, with minimal accuracy drop (< 0.3%).

Figures (10)

• Figure 1: Different possible execution strategies of a DNN on a Heterogeneous SoC with shared memory. Strategy A offloads the DNN computation to a single CU. Strategy B uses a layer-wise scheme where each layer can be computed by one of the available CU. Strategy C is the intra-layer mapping scheme enabled by ODiMO, where the computation of each layer can be split among the different CUs.
• Figure 2: Examples of $\theta$ encoding for Layer Selection (left) and for Mixed-Precision Assignment (right).
• Figure 3: General ODiMO mapping strategy for a layer.
• Figure 4: Final layer re-organization pass to support partitioning.
• Figure 5: Mappings obtained using ODiMO with latency as optimization target.