A Performance Evaluation of a Quantized Large Language Model on Various Smartphones

TL;DR

The paper investigates the feasibility of on-device LLM inference on recent iPhones using a quantized 7B Orca Mini model with 3-bit KS quantization in GGUF format. It benchmarks sampling, prompt decoding, and inference across five iPhone models under controlled thermal states to understand performance and thermal constraints. The results reveal hardware-specific variations, including a notable 20% uplift for the A17 Pro in sampling and anomalous low inference rates on the iPhone 15 Pro Max, highlighting driver/architecture influences. The study demonstrates that on-device LLMs on contemporary iPhones are feasible but demand improved power management, integration, and further compression/acceleration research to achieve robust, real-time performance in practice.

Abstract

This paper explores the feasibility and performance of on-device large language model (LLM) inference on various Apple iPhone models. Amidst the rapid evolution of generative AI, on-device LLMs offer solutions to privacy, security, and connectivity challenges inherent in cloud-based models. Leveraging existing literature on running multi-billion parameter LLMs on resource-limited devices, our study examines the thermal effects and interaction speeds of a high-performing LLM across different smartphone generations. We present real-world performance results, providing insights into on-device inference capabilities.

Figures (3)

• Figure 1: Sampling, prompt decoding, and inference results under different thermal states.
• Figure 2: Thermal state duration ratios for different iteration counts.
• Figure 3: Sampling, prompt decoding, and inference token rates of various iPhone models.