lm-Meter: Unveiling Runtime Inference Latency for On-Device Language Models

TL;DR

lm-Meter tackles the lack of lightweight, real-time profiling for on-device LLMs by introducing a phase- and kernel-level latency profiler that operates directly on mobile hardware with minimal overhead. The approach yields high-fidelity measurements across phases and kernels, enabling empirical studies of performance-accuracy trade-offs and identifying bottlenecks such as prefill and GPU-idle periods. It introduces the Harmonic Quantization score ($\mathcal{HQ}$) to holistically assess quantization effects across tasks and architectures. The findings demonstrate task- and architecture-dependent Pareto frontiers and reveal substantial opportunities for system- and model-level optimizations, thereby accelerating practical deployment of privacy-preserving, on-device LLMs.

Abstract

Large Language Models (LLMs) are increasingly integrated into everyday applications, but their prevalent cloud-based deployment raises growing concerns around data privacy and long-term sustainability. Running LLMs locally on mobile and edge devices (on-device LLMs) offers the promise of enhanced privacy, reliability, and reduced communication costs. However, realizing this vision remains challenging due to substantial memory and compute demands, as well as limited visibility into performance-efficiency trade-offs on resource-constrained hardware. We propose lm-Meter, the first lightweight, online latency profiler tailored for on-device LLM inference. lm-Meter captures fine-grained, real-time latency at both phase (e.g., embedding, prefill, decode, softmax, sampling) and kernel levels without auxiliary devices. We implement lm-Meter on commercial mobile platforms and demonstrate its high profiling accuracy with minimal system overhead, e.g., only 2.58% throughput reduction in prefill and 0.99% in decode under the most constrained Powersave governor. Leveraging lm-Meter, we conduct comprehensive empirical studies revealing phase- and kernel-level bottlenecks in on-device LLM inference, quantifying accuracy-efficiency trade-offs, and identifying systematic optimization opportunities. lm-Meter provides unprecedented visibility into the runtime behavior of LLMs on constrained platforms, laying the foundation for informed optimization and accelerating the democratization of on-device LLM systems. Code and tutorials are available at https://github.com/amai-gsu/LM-Meter.

Figures (11)

• Figure 1: Code snippet for querying OpenCL kernel‐execution timestamps.
• Figure 2: Phase‐level latency comparison between our lm-Meter and AGI ground-truth on Google Pixel 8 Pro running a quantized gemma-2-2b-it model (4 bit weights, 16 bit activations). Insets zoom in on the sub-millisecond segments, including Embedding, CopyProbsToCPU, and Sampling.
• Figure 3: Illustration of kernel duplication for fused_dequantize2_NT_matmul11, showing the original kernel (red) and its 50 duplicated instances (green). A broken x-axis with omitted rows is used to save space.
• Figure 4: Comparison of profiling-induced latency overhead, measured as throughput (tokens/s) during prefill and decode, under three CPU governors. Results compare lm-Meter, MELTing Point laskaridis2024melting, and a no-profiling baseline.
• Figure 5: Token-length distributions of each full dataset (blue) vs. KL-matched subset (orange). We sample 10% of ARC-Easy, ARC-Challenge, and GSM8K (238, 112, and 131 prompts) and 1% of HellaSwag (100 prompts). Reported KL divergences measure how closely each subset’s token-length histogram matches the full distribution.