Why does Prediction Accuracy Decrease over Time? Uncertain Positive Learning for Cloud Failure Prediction

TL;DR

The paper identifies Uncertain Positive Learning (UPLearning) as a real-world challenge arising during online model updating for cloud failure prediction, driven by mitigation actions that render some predicted failures unverifiable. It proposes Uptake, a generic risk-estimator that treats uncertain positives as both positive and negative during retraining, and demonstrates its effectiveness across public disk datasets and a large-scale Azure node dataset. Uptake consistently outperforms online updating with certain positives and approaches the offline upper bound, with stable gains across architectures (RNN, LSTM, Transformer, TCNN) and real production deployment. The work provides substantial practical impact by improving prediction accuracy and reliability in cloud systems where timely mitigation is crucial.

Abstract

With the rapid growth of cloud computing, a variety of software services have been deployed in the cloud. To ensure the reliability of cloud services, prior studies focus on failure instance (disk, node, and switch, etc.) prediction. Once the output of prediction is positive, mitigation actions are taken to rapidly resolve the underlying failure. According to our real-world practice in Microsoft Azure, we find that the prediction accuracy may decrease by about 9% after retraining the models. Considering that the mitigation actions may result in uncertain positive instances since they cannot be verified after mitigation, which may introduce more noise while updating the prediction model. To the best of our knowledge, we are the first to identify this Uncertain Positive Learning (UPLearning) issue in the real-world cloud failure prediction scenario. To tackle this problem, we design an Uncertain Positive Learning Risk Estimator (Uptake) approach. Using two real-world datasets of disk failure prediction and conducting node prediction experiments in Microsoft Azure, which is a top-tier cloud provider that serves millions of users, we demonstrate Uptake can significantly improve the failure prediction accuracy by 5% on average.

Figures (5)

• Figure 1: Toy example of cloud failure prediction model (binary classification for normal or failure instances) with offline updating and online updating. The prediction model (M) is updated by training with data in the dashed box in each time stage (T). The number on the top of each sub-figure illustrates the instance ID.
• Figure 2: Cloud failure prediction average F1-score over time.
• Figure 3: Example of positive, negative, and uncertain positive in the scenario of online cloud failure prediction.
• Figure 4: The average F1-score comparison of Uptake and its two compared approaches on three datasets: Alibaba, Backblaze, and Microsoft Azure. Each sub-figure corresponds to a specific type of base model. The error bar in the figure represents the upper and lower bounds of the results, indicating the robustness of Uptake across different base models.
• Figure 5: F1-score of Uptake under different $\pi_{\rm p}$ on two public datasets. The dots are the parameters chosen by our solution.