Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerating Depthwise Separable Convolutions on Ultra-Low-Power Devices

Francesco Daghero, Alessio Burrello, Massimo Poncino, Enrico Macii, Daniele Jahier Pagliari

TL;DR

This work tackles the challenge of efficiently accelerating depthwise separable convolutions on ultra-low-power edge devices by introducing six fused depthwise- and pointwise-kernel variants and an end-to-end deployment framework. By exploring multiple data-layout strategies and tiling schemes, the authors show substantial reductions in end-to-end latency (up to 11.40%) and in activation data movements (up to 52.97%) on the GAP8 SoC, along with meaningful reductions in memory transfers for several MobileNet-like networks. The approach is integrated into an open-source pipeline (DORY) with a fusion engine, tiler, and post-processing optimizer to select the best fusion strategy per network block, enabling effective deployment of DW-PW and PW-DW fused kernels on edge devices. The results emphasize the practical impact for edge AI, combining memory-hierarchy-aware kernel design with automated fusion selection to improve energy efficiency and responsiveness in IoT scenarios.

Abstract

Depthwise separable convolutions are a fundamental component in efficient Deep Neural Networks, as they reduce the number of parameters and operations compared to traditional convolutions while maintaining comparable accuracy. However, their low data reuse opportunities make deploying them notoriously difficult. In this work, we perform an extensive exploration of alternatives to fuse the depthwise and pointwise kernels that constitute the separable convolutional block. Our approach aims to minimize time-consuming memory transfers by combining different data layouts. When targeting a commercial ultra-low-power device with a three-level memory hierarchy, the GreenWaves GAP8 SoC, we reduce the latency of end-to-end network execution by up to 11.40%. Furthermore, our kernels reduce activation data movements between L2 and L1 memories by up to 52.97%.

Accelerating Depthwise Separable Convolutions on Ultra-Low-Power Devices

TL;DR

This work tackles the challenge of efficiently accelerating depthwise separable convolutions on ultra-low-power edge devices by introducing six fused depthwise- and pointwise-kernel variants and an end-to-end deployment framework. By exploring multiple data-layout strategies and tiling schemes, the authors show substantial reductions in end-to-end latency (up to 11.40%) and in activation data movements (up to 52.97%) on the GAP8 SoC, along with meaningful reductions in memory transfers for several MobileNet-like networks. The approach is integrated into an open-source pipeline (DORY) with a fusion engine, tiler, and post-processing optimizer to select the best fusion strategy per network block, enabling effective deployment of DW-PW and PW-DW fused kernels on edge devices. The results emphasize the practical impact for edge AI, combining memory-hierarchy-aware kernel design with automated fusion selection to improve energy efficiency and responsiveness in IoT scenarios.

Abstract

Depthwise separable convolutions are a fundamental component in efficient Deep Neural Networks, as they reduce the number of parameters and operations compared to traditional convolutions while maintaining comparable accuracy. However, their low data reuse opportunities make deploying them notoriously difficult. In this work, we perform an extensive exploration of alternatives to fuse the depthwise and pointwise kernels that constitute the separable convolutional block. Our approach aims to minimize time-consuming memory transfers by combining different data layouts. When targeting a commercial ultra-low-power device with a three-level memory hierarchy, the GreenWaves GAP8 SoC, we reduce the latency of end-to-end network execution by up to 11.40%. Furthermore, our kernels reduce activation data movements between L2 and L1 memories by up to 52.97%.
Paper Structure (18 sections, 1 equation, 4 figures, 2 tables)

This paper contains 18 sections, 1 equation, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background & Related Works
  3. Depthwise separable convolutions
  4. Depth-First Tiling & Layer-Fusion
  5. IoT Edge Nodes
  6. Materials & Methods
  7. Fused Kernel Design
  8. Depthwise-Pointwise
  9. Pointwise-Depthwise
  10. Data Layout Selection
  11. End-to-end Network Deployment
  12. Experimental Results
  13. Experimental setup
  14. Kernel analysis
  15. End-to-End Network Deployment
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Execution flow for standard and fused DW-PW sequences, for a layer with $FD=3$, and $IX/IY =8$.
  • Figure 2: Execution flow for standard and fused PW-DW sequences for a layer with $FD=4$ and $K=8$ with channel-wise (left) or row-wise (right) tiling.
  • Figure 3: Median execution cycles in L1 when changing the dimension of the intermediate buffer ($FD$).
  • Figure 4: Layer-by-layer execution cycles and memory transfers of the MV1-224 "Lowest Latency". The lighter colors show the improvement compared to the unfused execution.