ProfInfer: An eBPF-based Fine-Grained LLM Inference Profiler

TL;DR

ProfInfer tackles the challenge of obtaining fine-grained, low-overhead observability for on-device LLM inference by leveraging eBPF probes that attach non-intrusively to llama.cpp. It delivers three complementary views—ProfDAG, ProfTime, and ProfStat—mapping tokens, computation graphs, and operator-level behavior to hardware counter trends, enabling insights into MoE routing, KV-cache effects, and multi-backend offloading. The framework demonstrates low online overhead (under a few percent in many configurations) and supports adaptive tracing to meet QoS requirements, making performance profiling practical for optimization and resource-aware deployment on mobile/edge devices. By combining operator-semantics with hardware-aware tracing, ProfInfer provides actionable diagnostics for bottlenecks and scheduling, with concrete visualizations across token, graph, and operator granularity. This work advances on-device LLM profiling by offering non-intrusive, fine-grained observability that existing profilers typically lack.

Abstract

As large language models (LLMs) move from research to production, understanding how inference engines behave in real time has become both essential and elusive. Unlike general-purpose engines such as ONNX Runtime, today's LLM inference systems offer little operator-level visibility, leaving developers blind to where time and resources go. Even basic questions -- is this workload memory-bound or compute-bound? -- often remain unanswered. To close this gap, we develop a fine-grained, non-intrusive profiling framework for modern LLM inference engines, exemplified by llama-cpp but applicable to similar runtime architectures. Built on extended Berkeley Packet Filter (eBPF) technology, our system dynamically attaches probes to runtime functions across multiple layers -- without modifying or recompiling the source. It transforms collected traces into rich visualizations of operators, graphs, timelines, and hardware counter trends, exposing how dense inference, Mixture-of-Experts routing, and operator offloading behave in practice. With less than 4% runtime overhead and high profiling fidelity, our framework makes LLM inference both transparent and diagnosable, turning performance profiling into a practical tool for optimization, scheduling, and resource-aware deployment.

Figures (13)

• Figure 1: High-level design of ProfInfer comprising eBPF-based tracers followed by trace analyzers.
• Figure 2: An operational overview of ProfInfer's tracer for LLM inferences over llama.cpp. Basic metrics refer to type of the probe, time stamp, thread ID and CPU ID.
• Figure 3: Graphical view of an FFN for LLaMA-3.2-1B, in terms of memory bandwidth.
• Figure 4: Architecture differences in the self-attention of different LLMs.
• Figure 5: Timeline view in Perfetto. Only intra-operator parallelism is observed. (LLaMA3.2-1B-F16 with 4 threads.)