A Kernel-Based Approach for Accurate Steady-State Detection in Performance Time Series

TL;DR

The paper tackles the challenge of accurately identifying the transition from warm-up to steady state in performance time series, a critical step for reliable benchmarking. It introduces KB-KSSD, a kernel-based, online steady-state detector inspired by chemical reactor methods, combining smoothing, convolution-based step detection, and Kelly's steady-state testing within a windowed framework. Through a grid search and ground-truth grounding, KB-KSSD achieves about 14.5% lower total error than a state-of-the-art method, with fewer false negatives and more stable performance, indicating improved robustness and precision for diverse, noisy time series. The approach enhances benchmarking reliability and reproducibility, and its parameter sensitivity analysis demonstrates practical tunability, though future work is needed to address upward steps, kernel sizing, and broader applicability across benchmarks.

Abstract

This paper addresses the challenge of accurately detecting the transition from the warmup phase to the steady state in performance metric time series, which is a critical step for effective benchmarking. The goal is to introduce a method that avoids premature or delayed detection, which can lead to inaccurate or inefficient performance analysis. The proposed approach adapts techniques from the chemical reactors domain, detecting steady states online through the combination of kernel-based step detection and statistical methods. By using a window-based approach, it provides detailed information and improves the accuracy of identifying phase transitions, even in noisy or irregular time series. Results show that the new approach reduces total error by 14.5% compared to the state-of-the-art method. It offers more reliable detection of the steady-state onset, delivering greater precision for benchmarking tasks. For users, the new approach enhances the accuracy and stability of performance benchmarking, efficiently handling diverse time series data. Its robustness and adaptability make it a valuable tool for real-world performance evaluation, ensuring consistent and reproducible results.

Figures (14)

• Figure 1: The illustration of step detection via convolution with a large kernel.
• Figure 2: Number of (dis)agreements w.r.t. the larger dataset including manually-detected unsteady series. The number of total agreements denotes situations where both methods and the ground truth agree with each other, while the number of method agreements denotes the situation where both methods agree with each other, but not with the ground truth.
• Figure 3: Number of detected unsteady time series by both approaches w.r.t. the ground truth labels.
• Figure 4: Distribution of detection differences ( - ) for all time series. The plot also visualizes the range, where 95% of all errors stay within the vertical lines and there are Gaussians included to visualize an approximate normal distribution with the same $\mu$ and $\sigma$ to the real error distributions for easier readability.
• Figure 5: Detection differences ( - ) for all time series.