Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Scenario-Oriented Benchmark for Assessing AIOps Algorithms in Microservice Management

Yongqian Sun, Jiaju Wang, Zhengdan Li, Xiaohui Nie, Minghua Ma, Shenglin Zhang, Yuhe Ji, Lu Zhang, Wen Long, Hengmao Chen, Yongnan Luo, Dan Pei

TL;DR

MicroServo addresses the gap in AIOps evaluation by providing a live, scenario-oriented benchmark for microservice management that combines fault injection, real-time data collection, and containerized algorithm deployment. It defines a simple, unified dataset format for metrics, logs, and traces, and automates end-to-end evaluation across distinct fault scenarios, producing dynamic leaderboards tailored to specific operation contexts. The framework demonstrates three representative scenarios with open-source algorithms, illustrating how scenario specificity enhances algorithm selection and evaluation relevance. This approach offers practical benefits for practitioners seeking reproducible, real-time benchmarking and for researchers advancing AIOps methods in dynamic microservice environments.

Abstract

AIOps algorithms play a crucial role in the maintenance of microservice systems. Many previous benchmarks' performance leaderboard provides valuable guidance for selecting appropriate algorithms. However, existing AIOps benchmarks mainly utilize offline datasets to evaluate algorithms. They cannot consistently evaluate the performance of algorithms using real-time datasets, and the operation scenarios for evaluation are static, which is insufficient for effective algorithm selection. To address these issues, we propose an evaluation-consistent and scenario-oriented evaluation framework named MicroServo. The core idea is to build a live microservice benchmark to generate real-time datasets and consistently simulate the specific operation scenarios on it. MicroServo supports different leaderboards by selecting specific algorithms and datasets according to the operation scenarios. It also supports the deployment of various types of algorithms, enabling algorithms hot-plugging. At last, we test MicroServo with three typical microservice operation scenarios to demonstrate its efficiency and usability.

A Scenario-Oriented Benchmark for Assessing AIOps Algorithms in Microservice Management

TL;DR

MicroServo addresses the gap in AIOps evaluation by providing a live, scenario-oriented benchmark for microservice management that combines fault injection, real-time data collection, and containerized algorithm deployment. It defines a simple, unified dataset format for metrics, logs, and traces, and automates end-to-end evaluation across distinct fault scenarios, producing dynamic leaderboards tailored to specific operation contexts. The framework demonstrates three representative scenarios with open-source algorithms, illustrating how scenario specificity enhances algorithm selection and evaluation relevance. This approach offers practical benefits for practitioners seeking reproducible, real-time benchmarking and for researchers advancing AIOps methods in dynamic microservice environments.

Abstract

AIOps algorithms play a crucial role in the maintenance of microservice systems. Many previous benchmarks' performance leaderboard provides valuable guidance for selecting appropriate algorithms. However, existing AIOps benchmarks mainly utilize offline datasets to evaluate algorithms. They cannot consistently evaluate the performance of algorithms using real-time datasets, and the operation scenarios for evaluation are static, which is insufficient for effective algorithm selection. To address these issues, we propose an evaluation-consistent and scenario-oriented evaluation framework named MicroServo. The core idea is to build a live microservice benchmark to generate real-time datasets and consistently simulate the specific operation scenarios on it. MicroServo supports different leaderboards by selecting specific algorithms and datasets according to the operation scenarios. It also supports the deployment of various types of algorithms, enabling algorithms hot-plugging. At last, we test MicroServo with three typical microservice operation scenarios to demonstrate its efficiency and usability.
Paper Structure (14 sections, 3 figures, 7 tables)

This paper contains 14 sections, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Typical AIOps Tasks for Microservice Systems
  3. Anomaly Detection
  4. Failure Diagnosis
  5. System design
  6. Fault Injection
  7. Data Collection
  8. Algorithm Hot-plugging
  9. Evaluation Leaderboard
  10. Case Study
  11. Dataset
  12. Algorithms
  13. Results and Analysis
  14. Conclusion and Future Prospects

Figures (3)

  • Figure 1: Framework of MicroServo.
  • Figure 2: Structure of MicroServo-SDK.
  • Figure 3: Example of Evaluation Leaderboard.