Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerating Mobile Inference through Fine-Grained CPU-GPU Co-Execution

Zhuojin Li, Marco Paolieri, Leana Golubchik

TL;DR

This work tackles on-device neural inference latency by partitioning computation between CPU and GPU on mobile platforms with unified memory. It introduces kernel-implementation-aware latency predictors and a lightweight OpenCL fine-grained SVM-based synchronization to enable effective, low-overhead co-execution. Per-implementation predictors augmented with kernel-dispatch features enable accurate latency estimates, and end-to-end evaluations on four devices show speedups up to $1.89×$ for linear layers and $1.75×$ for convolutional layers, approaching brute-force optima of $2.01×$ and $1.87×$. The approach achieves dramatic reductions in synchronization overhead (to about $7 μs$) and demonstrates meaningful improvements on common architectures like VGG16 and ResNet variants, suggesting strong practical impact for mobile AI inference.

Abstract

Deploying deep neural networks on mobile devices is increasingly important but remains challenging due to limited computing resources. On the other hand, their unified memory architecture and narrower gap between CPU and GPU performance provide an opportunity to reduce inference latency by assigning tasks to both CPU and GPU. The main obstacles for such collaborative execution are the significant synchronization overhead required to combine partial results, and the difficulty of predicting execution times of tasks assigned to CPU and GPU (due to the dynamic selection of implementations and parallelism level). To overcome these obstacles, we propose both a lightweight synchronization mechanism based on OpenCL fine-grained shared virtual memory (SVM) and machine learning models to accurately predict execution times. Notably, these models capture the performance characteristics of GPU kernels and account for their dispatch times. A comprehensive evaluation on four mobile platforms shows that our approach can quickly select CPU-GPU co-execution strategies achieving up to 1.89x speedup for linear layers and 1.75x speedup for convolutional layers (close to the achievable maximum values of 2.01x and 1.87x, respectively, found by exhaustive grid search on a Pixel~5 smartphone).

Accelerating Mobile Inference through Fine-Grained CPU-GPU Co-Execution

TL;DR

This work tackles on-device neural inference latency by partitioning computation between CPU and GPU on mobile platforms with unified memory. It introduces kernel-implementation-aware latency predictors and a lightweight OpenCL fine-grained SVM-based synchronization to enable effective, low-overhead co-execution. Per-implementation predictors augmented with kernel-dispatch features enable accurate latency estimates, and end-to-end evaluations on four devices show speedups up to for linear layers and for convolutional layers, approaching brute-force optima of and . The approach achieves dramatic reductions in synchronization overhead (to about ) and demonstrates meaningful improvements on common architectures like VGG16 and ResNet variants, suggesting strong practical impact for mobile AI inference.

Abstract

Deploying deep neural networks on mobile devices is increasingly important but remains challenging due to limited computing resources. On the other hand, their unified memory architecture and narrower gap between CPU and GPU performance provide an opportunity to reduce inference latency by assigning tasks to both CPU and GPU. The main obstacles for such collaborative execution are the significant synchronization overhead required to combine partial results, and the difficulty of predicting execution times of tasks assigned to CPU and GPU (due to the dynamic selection of implementations and parallelism level). To overcome these obstacles, we propose both a lightweight synchronization mechanism based on OpenCL fine-grained shared virtual memory (SVM) and machine learning models to accurately predict execution times. Notably, these models capture the performance characteristics of GPU kernels and account for their dispatch times. A comprehensive evaluation on four mobile platforms shows that our approach can quickly select CPU-GPU co-execution strategies achieving up to 1.89x speedup for linear layers and 1.75x speedup for convolutional layers (close to the achievable maximum values of 2.01x and 1.87x, respectively, found by exhaustive grid search on a Pixel~5 smartphone).
Paper Structure (15 sections, 1 equation, 7 figures, 4 tables)

This paper contains 15 sections, 1 equation, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Inference Workload Partitioning
  3. ML Operation Partitioning.
  4. Problem Formulation.
  5. Accurate Latency Prediction
  6. GPU Kernel Characterization
  7. Feature Augmentation
  8. Reduction of CPU-GPU Synchronization Overhead
  9. Experimental Results
  10. Experimental Setup
  11. Training Dataset Generation and Latency Prediction Accuracy
  12. Co-Execution Speedup of Individual Layers
  13. Co-Execution Speedup of End-to-End Models
  14. Ablation Study
  15. Conclusions

Figures (7)

  • Figure 1: Introduction to Mobile Platforms
  • Figure 2: Comparison of CPU and GPU Latencies for Linear Operations with Input Shape (50, 3072), with 95% Confidence Intervals (OnePlus 11)
  • Figure 3:
  • Figure 4: Illustration of Computation Partitioning over Output Channels
  • Figure 5: Latency Prediction Improvement using Additional Features (OnePlus 11)
  • ...and 2 more figures