Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PICO: Pipeline Inference Framework for Versatile CNNs on Diverse Mobile Devices

Xiang Yang, Zikang Xu, Qi Qi, Jingyu Wang, Haifeng Sun, Jianxin Liao, Song Guo

TL;DR

PICO tackles the challenge of distributed CNN inference on heterogeneous mobile devices by transforming complex CNN graphs into a pipeline of stages and solving a many-to-many device assignment. It introduces a graph-partitioning algorithm to produce fine-grained pieces and a dynamic-programming-based approach to configure stages, adapting to heterogeneity with greedy refinements. The framework reduces redundant computation and cross-device communication, achieving substantial throughput gains (up to $6.8\times$ in experiments with Raspberry-Pi clusters) and lower memory/energy footprints compared to baselines. This approach enables scalable, real-time CNN inference on edge devices without retraining, offering practical benefits for smart cameras, edge AI, and distributed vision tasks.

Abstract

Distributing the inference of convolutional neural network (CNN) to multiple mobile devices has been studied in recent years to achieve real-time inference without losing accuracy. However, how to map CNN to devices remains a challenge. On the one hand, scheduling the workload of state-of-the-art CNNs with multiple devices is NP-Hard because the structures of CNNs are directed acyclic graphs (DAG) rather than simple chains. On the other hand, distributing the inference workload suffers from expensive communication and unbalanced computation due to the wireless environment and heterogeneous devices. This paper presents PICO, a pipeline cooperation framework to accelerate the inference of versatile CNNs on diverse mobile devices. At its core, PICO features: (1) a generic graph partition algorithm that considers the characteristics of any given CNN and orchestrates it into a list of model pieces with suitable granularity, and (2) a many-to-many mapping algorithm that produces the best pipeline configuration for heterogeneous devices. In our experiment with 2 ~ 8 Raspberry-Pi devices, the throughput can be improved by 1.8 ~ 6.8x under different CPU frequencies.

PICO: Pipeline Inference Framework for Versatile CNNs on Diverse Mobile Devices

TL;DR

PICO tackles the challenge of distributed CNN inference on heterogeneous mobile devices by transforming complex CNN graphs into a pipeline of stages and solving a many-to-many device assignment. It introduces a graph-partitioning algorithm to produce fine-grained pieces and a dynamic-programming-based approach to configure stages, adapting to heterogeneity with greedy refinements. The framework reduces redundant computation and cross-device communication, achieving substantial throughput gains (up to in experiments with Raspberry-Pi clusters) and lower memory/energy footprints compared to baselines. This approach enables scalable, real-time CNN inference on edge devices without retraining, offering practical benefits for smart cameras, edge AI, and distributed vision tasks.

Abstract

Distributing the inference of convolutional neural network (CNN) to multiple mobile devices has been studied in recent years to achieve real-time inference without losing accuracy. However, how to map CNN to devices remains a challenge. On the one hand, scheduling the workload of state-of-the-art CNNs with multiple devices is NP-Hard because the structures of CNNs are directed acyclic graphs (DAG) rather than simple chains. On the other hand, distributing the inference workload suffers from expensive communication and unbalanced computation due to the wireless environment and heterogeneous devices. This paper presents PICO, a pipeline cooperation framework to accelerate the inference of versatile CNNs on diverse mobile devices. At its core, PICO features: (1) a generic graph partition algorithm that considers the characteristics of any given CNN and orchestrates it into a list of model pieces with suitable granularity, and (2) a many-to-many mapping algorithm that produces the best pipeline configuration for heterogeneous devices. In our experiment with 2 ~ 8 Raspberry-Pi devices, the throughput can be improved by 1.8 ~ 6.8x under different CPU frequencies.
Paper Structure (51 sections, 5 theorems, 16 equations, 18 figures, 7 tables, 3 algorithms)

This paper contains 51 sections, 5 theorems, 16 equations, 18 figures, 7 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background And Motivation
  3. Procedure of CNN Inference
  4. Parallelizing CNNs With Mobile Devices
  5. The Structures Of CNNs
  6. Motivation And Pipeline Inference
  7. Motivation
  8. Pipeline Inference
  9. System Model
  10. Problem Define
  11. CNN With Graph Structure
  12. Optimization Goal
  13. Cost Model
  14. The Input Feature Size For Devices
  15. Inference Cost Of Devices
  16. ...and 36 more sections

Key Result

Theorem 1

Given a CNN model $\mathbb{G}$ with chain structure, the problem of minimizing maximum stage execution time $\mathcal{P}(\mathbb{G}, \mathbb{D}, \mathbb{S})$ with heterogeneous mobile devices under a constriction that $\mathcal{T}(\mathbb{G}, \mathbb{D}, \mathbb{S}) \leq T_{lim}$ is NP-hard.

Figures (18)

  • Figure 1: A diagrammatic sketch of pipeline inference.
  • Figure 2: The communication and computation percentages of each layer.
  • Figure 3: CNNs with different structures: The chain structure is the simplest one which just put the neural layers into a sequence. Block structure replaces the element in chain structure from neural layer to block, each block can be seen as a directed acyclic graph (DAG). Graph structure can not be partitioned into blocks, the entire structure is a huge DAG.
  • Figure 4: Feature map partition strategy introduces redundant calculation.
  • Figure 5: Computation overhead with different partition settings.
  • ...and 13 more figures

Theorems & Definitions (14)

  • Definition 1
  • Definition 2
  • Definition 3
  • Theorem 1
  • Proof 3.1
  • Theorem 2
  • Proof 3.2
  • Definition 4
  • Definition 5
  • Theorem 3
  • ...and 4 more