Long-term Monitoring of Kernel and Hardware Events to Understand Latency Variance

TL;DR

VarMRI offers a practical framework for long-term monitoring of kernel and hardware events that influence latency variance, addressing the big-data challenge with selective, thread-granularity recording and cumulative-value counters. Its offline impact-value analysis, supported by two domain-specific rules for causal inference, provides actionable prioritization of events to optimize tail latencies. The study presents a 3,000-hour CloudLab evaluation across six apps, showing tail-latency reductions up to about $31\%$ and notable cross-experiment variability, underscoring the need for long-term monitoring. By combining a low-overhead online recorder with robust offline analysis, VarMRI enables systematic diagnosis of latency variance while remaining compatible with existing application-level tracing, with potential extensions to more architectures and hardware domains.

Abstract

This paper presents our experience to understand latency variance caused by kernel and hardware events, which are often invisible at the application level. For this purpose, we have built VarMRI, a tool chain to monitor and analyze those events in the long term. To mitigate the "big data" problem caused by long-term monitoring, VarMRI selectively records a subset of events following two principles: it only records events that are affecting the requests recorded by the application; it records coarse-grained information first and records additional information only when necessary. Furthermore, VarMRI introduces an analysis method that is efficient on large amount of data, robust on different data set and against missing data, and informative to the user. VarMRI has helped us to carry out a 3,000-hour study of six applications and benchmarks on CloudLab. It reveals a wide variety of events causing latency variance, including interrupt preemption, Java GC, pipeline stall, NUMA balancing etc.; simple optimization or tuning can reduce tail latencies by up to 31%. Furthermore, the impacts of some of these events vary significantly across different experiments, which confirms the necessity of long-term monitoring.

Figures (10)

• Figure 1: The architecture of VarMRI.
• Figure 2: Usage of VarMRI API.
• Figure 3: An example of executing multiple loops (loop1 and loop3 run on thread T1; loop2 runs on thread T2; loop2 and loop3 are selected for recording).
• Figure 4: Recording the lengths of interrupts and faults.
• Figure 5: Loop bench experiments.

Theorems & Definitions (2)

• Definition 1
• Definition 2