Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The PetShop Dataset -- Finding Causes of Performance Issues across Microservices

Michaela Hardt, William R. Orchard, Patrick Blöbaum, Shiva Kasiviswanathan, Elke Kirschbaum

TL;DR

The paper introduces the PetShop Dataset as a standardized benchmark for root-cause analysis in microservice-based applications, addressing the lack of public datasets for quantitative RCA evaluation. It provides 41 components, 68 injected issues, and ground-truth root causes across latency, requests, and availability measured every 5 minutes, with train/test splits and an AWS-X-Ray-based service map. By benchmarking a range of RCA methods across causal and non-causal formulations, the authors show that access to a causal graph improves performance, but SCM-based approaches struggle when data are limited and simple correlation baselines remain competitive. The dataset, accompanying tooling, and evaluation protocol are publicly available to accelerate robust RCA research and foster community contributions across different applications.

Abstract

Identifying root causes for unexpected or undesirable behavior in complex systems is a prevalent challenge. This issue becomes especially crucial in modern cloud applications that employ numerous microservices. Although the machine learning and systems research communities have proposed various techniques to tackle this problem, there is currently a lack of standardized datasets for quantitative benchmarking. Consequently, research groups are compelled to create their own datasets for experimentation. This paper introduces a dataset specifically designed for evaluating root cause analyses in microservice-based applications. The dataset encompasses latency, requests, and availability metrics emitted in 5-minute intervals from a distributed application. In addition to normal operation metrics, the dataset includes 68 injected performance issues, which increase latency and reduce availability throughout the system. We showcase how this dataset can be used to evaluate the accuracy of a variety of methods spanning different causal and non-causal characterisations of the root cause analysis problem. We hope the new dataset, available at https://github.com/amazon-science/petshop-root-cause-analysis/ enables further development of techniques in this important area.

The PetShop Dataset -- Finding Causes of Performance Issues across Microservices

TL;DR

The paper introduces the PetShop Dataset as a standardized benchmark for root-cause analysis in microservice-based applications, addressing the lack of public datasets for quantitative RCA evaluation. It provides 41 components, 68 injected issues, and ground-truth root causes across latency, requests, and availability measured every 5 minutes, with train/test splits and an AWS-X-Ray-based service map. By benchmarking a range of RCA methods across causal and non-causal formulations, the authors show that access to a causal graph improves performance, but SCM-based approaches struggle when data are limited and simple correlation baselines remain competitive. The dataset, accompanying tooling, and evaluation protocol are publicly available to accelerate robust RCA research and foster community contributions across different applications.

Abstract

Identifying root causes for unexpected or undesirable behavior in complex systems is a prevalent challenge. This issue becomes especially crucial in modern cloud applications that employ numerous microservices. Although the machine learning and systems research communities have proposed various techniques to tackle this problem, there is currently a lack of standardized datasets for quantitative benchmarking. Consequently, research groups are compelled to create their own datasets for experimentation. This paper introduces a dataset specifically designed for evaluating root cause analyses in microservice-based applications. The dataset encompasses latency, requests, and availability metrics emitted in 5-minute intervals from a distributed application. In addition to normal operation metrics, the dataset includes 68 injected performance issues, which increase latency and reduce availability throughout the system. We showcase how this dataset can be used to evaluate the accuracy of a variety of methods spanning different causal and non-causal characterisations of the root cause analysis problem. We hope the new dataset, available at https://github.com/amazon-science/petshop-root-cause-analysis/ enables further development of techniques in this important area.
Paper Structure (22 sections, 2 equations, 6 figures, 9 tables)

This paper contains 22 sections, 2 equations, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Related Work
  4. RCA in microservice-based applications
  5. RCA for other applications
  6. Dataset Description
  7. Data Structure
  8. Service Map
  9. Service-level Objectives (SLOs)
  10. Traffic Scenarios
  11. Metrics
  12. Injected Issues
  13. Evaluation
  14. Experiments
  15. Limitations
  16. ...and 7 more sections

Figures (6)

  • Figure 1: Overview of pet adoption site built on microservices. We injected issues at the highlighted nodes that cause SLO violations of high latencies and low availability at the PetSide node.
  • Figure 2: Exemplary printout of the metrics for two microservices with 5 measurements each. The multi-column format contains the node, the metric, and different statistics for the aggregation of the metric over 5 minute time windows.
  • Figure 3: Example showing how delays in the payforadoption ECS Fargate microservice reflect in the response times at the PetSite.
  • Figure 4: Screenshot of the landing webpage of the pet adoption application.
  • Figure 5: Distribution of IT anomaly scores at PetSite for latency issues.
  • ...and 1 more figures

Theorems & Definitions (1)

  • Example 1