Can We Recycle Our Old Models? An Empirical Evaluation of Model Selection Mechanisms for AIOps Solutions

TL;DR

This work investigates whether selecting among historical AIOps models can outperform periodic retraining under concept drift. It introduces six model selection mechanisms (TBM, rTBM, SBM, rSBM, CRC, LaF) plus baselines and an oracle, and evaluates them on three large public datasets (Google cluster trace, Backblaze disk stats, Alibaba GPU cluster trace) using AUC for prediction and rank-based metrics (Kendall's tau, Jaccard) to assess model rankings and stability. The findings show that temporal-adjacency based selections, particularly rSBM, often achieve higher predictive performance and better alignment with the oracle, though a gap remains between current methods and the theoretical upper bound. These results suggest practical value in reusing historical models with principled selection to improve robustness to concept drift in AIOps, and point to future work on approaches that close the remaining gap to optimal performance and integrate ranking signals into ensemble strategies.

Abstract

AIOps (Artificial Intelligence for IT Operations) solutions leverage the tremendous amount of data produced during the operation of large-scale systems and machine learning models to assist software practitioners in their system operations. Existing AIOps solutions usually maintain AIOps models against concept drift through periodical retraining, despite leaving a pile of discarded historical models that may perform well on specific future data. Other prior works propose dynamically selecting models for prediction tasks from a set of candidate models to optimize the model performance. However, there is no prior work in the AIOps area that assesses the use of model selection mechanisms on historical models to improve model performance or robustness. To fill the gap, we evaluate several model selection mechanisms by assessing their capabilities in selecting the optimal AIOps models that were built in the past to make predictions for the target data. We performed a case study on three large-scale public operation datasets: two trace datasets from the cloud computing platforms of Google and Alibaba, and one disk stats dataset from the BackBlaze cloud storage data center. We observe that the model selection mechnisms utilizing temporal adjacency tend to have a better performance and can prevail the periodical retraining approach. Our findings also highlight a performance gap between existing model selection mechnisms and the theoretical upper bound which may motivate future researchers and practitioners in investigating more efficient and effective model selection mechanisms that fit in the context of AIOps.

Figures (9)

• Figure 1: Data schema for our studied datasets. Each colored box represents a data table: a line of the table name followed by lines describing the data fields. For the Google and Alibaba datasets, each table (e.g., machine_events) is one or multiple CSV files containing the fields described in the box. For the Backblaze dataset, the tables represent the logical view, while the physical data is stored as daily snapshots of each disk's attributes.
• Figure 2: The average AUC performance of model selection mechanisms in each testing period.
• Figure 3: Scott-Knott test results of the AUC performance from different multi-model selection mechanisms.
• Figure 4: The average Kendall's $\tau$ correlation between model selection mechanisms and the oracle ranking in each testing period.
• Figure 5: Scott-Knott test results of the Kendall's $\tau$ correlation from different multi-model selection mechanisms. The horizontal lines indicate the threshold for interpreting ranking agreement.