LO2: Microservice API Anomaly Dataset of Logs and Metrics

TL;DR

LO2 provides a first large-scale multi-modal dataset for anomaly detection in microservices by collecting logs and metrics from a production OSS system (Light-OAuth2) under Locust-driven API testing. Although traces are incomplete due to instrumentation limits, the dataset includes ~657k log files and >45 million metric files across 1740 runs, plus metadata and replication scripts to enable FAIR reuse. Preliminary analyses show that log-based anomaly detection benefits from per-service modeling with a DecisionTree approach, while metric-based PCA identifies a compact set of memory and disk features driving variance. The work offers a valuable resource for multi-modal fusion research in production MSS and outlines concrete paths for extending modalities, improving tracing, and enabling broader community contributions.

Abstract

Context. Microservice-based systems have gained significant attention over the past years. A critical factor for understanding and analyzing the behavior of these systems is the collection of monitoring data such as logs, metrics, and traces. These data modalities can be used for anomaly detection and root cause analysis of failures. In particular, multi-modal methods utilizing several types of this data at once have gained traction in the research community since these three modalities capture different dimensions of system behavior. Aim. We provide a dataset that supports research on anomaly detection and architectural degradation in microservice systems. We generate a comprehensive dataset of logs, metrics, and traces from a production microservice system to enable the exploration of multi-modal fusion methods that integrate multiple data modalities. Method. We dynamically tested the various APIs of the MS-based system, implementing the OAuth2.0 protocol using the Locust tool. For each execution of the prepared test suite, we collect logs and performance metrics for correct and erroneous calls with data labeled according to the error triggered during the call. Contributions. We collected approximately 657,000 individual log files, totaling over two billion log lines. In addition, we collected more than 45 million individual metric files that contain 485 unique metrics. We provide an initial analysis of logs, identify key metrics through PCA, and discuss challenges in collecting traces for this system. Moreover, we highlight the possibilities for making a more fine-grained version of the data set. This work advances anomaly detection in microservice systems using multiple data sources.

Figures (2)

• Figure 1: Data Collection Process
• Figure 2: Number of Log Lines per Type and Service