Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inference

Haolin Zhang, Jeff Huang

TL;DR

This work challenges the default assumption that GPUs are universally superior for on-device LLM inference. By evaluating six models (0.5B–8B) on an iPhone 15 Pro with llama.cpp across CPU and GPU backends and multiple precisions, it shows that carefully tuned CPU inference can exceed GPU throughput for smaller models, driven by reduced kernel overheads and architecture-aware threading. The dominant bottleneck is GEMM in the FFN, with Q4 quantization offering substantial speedups, while larger models still favor GPUs due to memory and compute demands. The study also introduces a graph-aware topological execution approach, demonstrates its potential and limitations, and highlights profiling gaps on mobile hardware that hinder deeper attribution of bottlenecks. Together, these findings argue for hardware-aware, hybrid deployment strategies and targeted GEMM optimizations to maximize on-device LLM performance in constrained environments.

Abstract

The common assumption in on-device AI is that GPUs, with their superior parallel processing, always provide the best performance for large language model (LLM) inference. In this work, we challenge this notion by empirically demonstrating that, under certain conditions, CPUs can outperform GPUs for LLM inference on mobile devices. Using a 1-billion-parameter LLM deployed via llama.cpp on the iPhone 15 Pro, we show that a CPU-only configuration (two threads, F16 precision) achieves 17 tokens per second, surpassing the 12.8 tokens per second obtained with GPU acceleration. We analyze the architectural factors driving this counterintuitive result, revealing that GPU memory transfer overhead and CPU thread optimization play a critical role. Furthermore, we explore the impact of thread oversubscription, quantization strategies, and hardware constraints, providing new insights into efficient on-device AI execution. Our findings challenge conventional GPU-first thinking, highlighting the untapped potential of optimized CPU inference and paving the way for smarter deployment strategies in mobile AI. However, fully explaining the observed CPU advantage remains difficult due to limited access to low-level profiling tools on iOS.

Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inference

TL;DR

This work challenges the default assumption that GPUs are universally superior for on-device LLM inference. By evaluating six models (0.5B–8B) on an iPhone 15 Pro with llama.cpp across CPU and GPU backends and multiple precisions, it shows that carefully tuned CPU inference can exceed GPU throughput for smaller models, driven by reduced kernel overheads and architecture-aware threading. The dominant bottleneck is GEMM in the FFN, with Q4 quantization offering substantial speedups, while larger models still favor GPUs due to memory and compute demands. The study also introduces a graph-aware topological execution approach, demonstrates its potential and limitations, and highlights profiling gaps on mobile hardware that hinder deeper attribution of bottlenecks. Together, these findings argue for hardware-aware, hybrid deployment strategies and targeted GEMM optimizations to maximize on-device LLM performance in constrained environments.

Abstract

The common assumption in on-device AI is that GPUs, with their superior parallel processing, always provide the best performance for large language model (LLM) inference. In this work, we challenge this notion by empirically demonstrating that, under certain conditions, CPUs can outperform GPUs for LLM inference on mobile devices. Using a 1-billion-parameter LLM deployed via llama.cpp on the iPhone 15 Pro, we show that a CPU-only configuration (two threads, F16 precision) achieves 17 tokens per second, surpassing the 12.8 tokens per second obtained with GPU acceleration. We analyze the architectural factors driving this counterintuitive result, revealing that GPU memory transfer overhead and CPU thread optimization play a critical role. Furthermore, we explore the impact of thread oversubscription, quantization strategies, and hardware constraints, providing new insights into efficient on-device AI execution. Our findings challenge conventional GPU-first thinking, highlighting the untapped potential of optimized CPU inference and paving the way for smarter deployment strategies in mobile AI. However, fully explaining the observed CPU advantage remains difficult due to limited access to low-level profiling tools on iOS.
Paper Structure (34 sections, 10 figures, 1 algorithm)

This paper contains 34 sections, 10 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Advancements in Small Language Models
  4. On-Device Deployment of SLMs
  5. Measurement and Survey of On-Device LLM Deployment
  6. Background
  7. Methodology
  8. Hardware and Software Setup
  9. Model Selection and Precision
  10. Test Configurations
  11. Experimental Procedure
  12. Evaluation Metrics
  13. Results
  14. Performance Comparison Across Models and Backends
  15. Memory and Timeout Constraints.
  16. ...and 19 more sections

Figures (10)

  • Figure 1: Segment of the compute graph for a LLaMA 3.2 model.
  • Figure 2: Left (GPU-enabled): Nodes are offloaded to the GPU backend. Right (CPU-only execution): Nodes are processed in parallel across four CPU threads.
  • Figure 3: Execution sequence of nodes within transformer blocks. Each operation is processed sequentially.
  • Figure 4: Performance Comparison of Different Models Across Backends
  • Figure 5: Profiling result of different OPs in prefill and decode stages.
  • ...and 5 more figures