Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inference

TL;DR

This work tackles the challenge of accelerating large language model inference on mobile devices by exploiting the combined power of GPU and NPU on heterogeneous SoCs. It introduces HeteroInfer, a runtime that integrates layer-level and tensor-level GPU-NPU parallelism, a fast synchronization mechanism, and an offline profiler–solver pipeline to tailor execution to device characteristics. The approach yields end-to-end speedups from 1.34$ imes$ to 6.02$ imes$ over GPU-only and NPU-only baselines, with substantial gains in prefill and decoding phases and minimal interference with other apps. The results demonstrate the practicality of efficient edge-LMM inference on mainstream mobile hardware and offer design insights for future edge AI accelerators and systems.

Abstract

With the rapid advancement of artificial intelligence technologies such as ChatGPT, AI agents, and video generation, contemporary mobile systems have begun integrating these AI capabilities on local devices to enhance privacy and reduce response latency. To meet the computational demands of AI tasks, current mobile SoCs are equipped with diverse AI accelerators, including GPUs and Neural Processing Units (NPUs). However, there has not been a comprehensive characterization of these heterogeneous processors, and existing designs typically only leverage a single AI accelerator for LLM inference, leading to suboptimal use of computational resources and memory bandwidth. In this paper, we first summarize key performance characteristics of heterogeneous processors, SoC memory bandwidth, etc. Drawing on these observations, we propose different heterogeneous parallel mechanisms to fully exploit both GPU and NPU computational power and memory bandwidth. We further design a fast synchronization mechanism between heterogeneous processors that leverages the unified memory architecture. By employing these techniques, we present HeteroInfer, the fastest LLM inference engine in mobile devices which supports GPU-NPU heterogeneous execution. Evaluation shows that HeteroInfer delivers a 1.34x to 6.02x end-to-end speedup over state-of-the-art GPU-only and NPU-only LLM engines, while maintaining negligible interference with other applications.

Figures (20)

• Figure 1: The GPU performance with varying tensor sizes.
• Figure 2: The hardware design for NPU: Systolic array with the weight stall computing paradigm.
• Figure 3: The stage performance of NPUs. The execution time of the Matmul operator across different tensor sizes.
• Figure 4: The order-sensitive and shape-sensitive performance of NPUs. The performance of the NPU is significantly influenced by the order and shape of the tensors.
• Figure 5: The total memory bandwidth with single and multiple processors. We execute the decoding workloads across different backends and measure the available memory bandwidth in the entire SoC.