RooflineBench: A Benchmarking Framework for On-Device LLMs via Roofline Analysis

TL;DR

RooflineBench adopts the Roofline model to quantify on-device LLM inference potential, introducing the Relative Inference Potential ($\Phi$) to compare efficiency across heterogeneous hardware. By linking FLOPs, memory traffic, and empirical hardware ceilings, the approach reveals regime transitions driven by context length and model depth, and demonstrates how architectural refinements like Multi-head Latent Attention (MLA) shift execution toward the compute-bound region. Key findings show a non-monotonic $OI$ with depth, significant hardware-structure interactions (an efficiency trap), and robust cross-platform gains from MLA; the work provides actionable guidance for hardware-software co-design in edge AI. The framework enables fair, hardware-aware benchmarking beyond raw throughput, helping align neural architectures with physical constraints for localized intelligence.

Abstract

The transition toward localized intelligence through Small Language Models (SLMs) has intensified the need for rigorous performance characterization on resource-constrained edge hardware. However, objectively measuring the theoretical performance ceilings of diverse architectures across heterogeneous platforms remains a formidable challenge. In this work, we propose a systematic framework based on the Roofline model that unifies architectural primitives and hardware constraints through the lens of operational intensity (OI). By defining an inference-potential region, we introduce the Relative Inference Potential as a novel metric to compare efficiency differences between Large Language Models (LLMs) on the same hardware substrate. Extensive empirical analysis across diverse compute tiers reveals that variations in performance and OI are significantly influenced by sequence length. We further identify a critical regression in OI as model depth increases. Additionally, our findings highlight an efficiency trap induced by hardware heterogeneity and demonstrate how structural refinements, such as Multi-head Latent Attention (M LA), can effectively unlock latent inference potential across various hardware substrates. These insights provide actionable directions for hardware-software co-design to align neural structures with physical constraints in on-device intelligence. The released code is available in the Appendix C.

Figures (25)

• Figure 1: Overview of the RooflineBench framework and analytical methodology. Left: The integrated profiling system incorporating on-device LLM configurations and hardware specifications for systematic analysis. Right: Graphical representation of the Roofline model illustrating the Relative Inference Potential between observed throughput and theoretical hardware ceilings across both memory-bound and compute-bound regimes.
• Figure 2: Comparison of roofline models across different input-output sequence length scenarios on Apple M1 Pro. Performance is evaluated in FP16 precision for Qwen2.5-Instruct (GQA), Llama-3.2-Instruct (GQA), PLM-Instruct (MLA), and SmolLM2-Instruct (MHA). The markers represent four inference scenarios: SISO, SILO, LISO, and LILO. Detailed analysis of the sensitivity to attention mechanisms is provided in subsequent sections. For comprehensive and detailed benchmarks across a wider range of hardware devices, please refer to Appendix \ref{['sec:Appendix D Benchmark Results']}.
• Figure 3: The figure shows the inference performance of Qwen2.5-Instruct FP16, Q8_0, and Q4_K_M with 2 to 64 layers (Experimental platform: Apple M1 Pro). Comprehensive and detailed benchmarks across a wider range of hardware devices are in Appendix \ref{['sec:Appendix D Benchmark Results']}.
• Figure 4: Performance comparison of various models across three different precision settings and all benchmarked hardware platforms. The results demonstrate a consistent performance gap and uniform trends, regardless of the specific device or precision level employed.
• Figure 5: Analysis of hardware utilization efficiency and architectural scalability across heterogeneous platforms. The left panel presents the Theoretical Roofline Comparison for five representative devices including RTX 3090, RTX 3070Ti Laptop, Apple M1 Pro, Jetson Orin Nano, and Raspberry Pi 5, illustrating a significant disparity in theoretical ridge points ranging from 8.98 to 38.00 $FLOPs/Byte$. The right panel depicts the Measured Performance in the LISO scenario for MLA, MHA, and GQA configurations across the Apple M1 Pro and RTX 3070Ti Laptop, highlighting the consistent efficiency baseline and parallel shift in operational intensity and performance when scaling across hardware tiers.