eBPF-Based Instrumentation for Generalisable Diagnosis of Performance Degradation
Diogo Landau, Jorge Barbosa, Nishant Saurabh
TL;DR
The paper addresses QoS degradation in online data-intensive apps by introducing $16$ fine-grained eBPF-based metrics that span six kernel subsystems, enabling application-agnostic diagnosis via selective, per-thread tracking. It presents a practical design and architecture for instrumenting kernel interactions (scheduling, VFS, networking, futex, IO multiplexing, and disk IO) with a Rust-based collector that aggregates per-second summaries to minimize overhead. Through extensive experiments on seven diverse applications (databases, Kafka, Solr, Teastore, Redis, and ML-inference), the approach demonstrates its ability to deconstruct degradation into concrete causes such as disk contention, lock contention, CPU saturation, and external service bottlenecks, while highlighting modest instrumentation overhead. The work contributes a reproducible framework and an extensible set of metrics for real-world performance debugging, with future plans to incorporate memory contention analyses and broaden applicability beyond Linux systems.
Abstract
Online Data Intensive applications (e.g. message brokers, ML inference and databases) are core components of the modern internet, providing critical functionalities to connecting services. The load variability and interference they experience are generally the main causes of Quality of Service (QoS) degradation, harming depending applications, and resulting in an impaired end-user experience. Uncovering the cause of QoS degradation requires detailed instrumentation of an application's activity. Existing generalisable approaches utilise readily available system metrics that encode interference in kernel metrics, but unfortunately, these approaches lack the required detail to pinpoint granular causes of performance degradation (e.g., lock, disk and CPU contention). In contrast, this paper explores the use of fine-grained system-level metrics to facilitate an application-agnostic diagnosis of QoS degradation. To this end, we introduce and implement $16$ $\textit{eBPF-based metrics}$ spanning over six kernel subsystems, which capture statistics over kernel events that often highlight obstacles impeding an application's progress. We demonstrate the use of our $\textit{eBPF-based metrics}$ through extensive experiments containing a representative set of online data-intensive applications. Results show that the implemented metrics can deconstruct performance degradation when applications face variable workload patterns and common resource contention scenarios, while also revealing applications' internal architecture constraints.