Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AlertGuardian: Intelligent Alert Life-Cycle Management for Large-scale Cloud Systems

Guangba Yu, Genting Mai, Rui Wang, Ruipeng Li, Pengfei Chen, Long Pan, Ruijie Xu

TL;DR

The paper tackles alert fatigue in large-scale cloud systems by introducing AlertGuardian, a lifecycle-aware framework that combines a lightweight graph-based denoising module with Retrieval-Augmented Generation (RAG) powered alert summarization and an offline, multi-agent rule refinement workflow. The approach addresses three core gaps: online noise filtering, contextual alert summaries, and iterative rule improvement, validated on four real-world datasets with substantial reductions in alert volume and high-quality diagnostic narratives. Results show roughly 94–95% alert reduction, 98.5% action accuracy in summaries, and 31–33% rule acceptance across datasets, underscoring practical impact on MTTR and operator workload. The work demonstrates how a hybrid of efficient graph models and LLMs, reinforced by RAG and human-in-the-loop validation, can sustainably improve alert management in dynamic cloud environments, with concrete deployment lessons and guidelines for future research.

Abstract

Alerts are critical for detecting anomalies in large-scale cloud systems, ensuring reliability and user experience. However, current systems generate overwhelming volumes of alerts, degrading operational efficiency due to ineffective alert life-cycle management. This paper details the efforts of Company-X to optimize alert life-cycle management, addressing alert fatigue in cloud systems. We propose AlertGuardian, a framework collaborating large language models (LLMs) and lightweight graph models to optimize the alert life-cycle through three phases: Alert Denoise uses graph learning model with virtual noise to filter noise, Alert Summary employs Retrieval Augmented Generation (RAG) with LLMs to create actionable summary, and Alert Rule Refinement leverages multi-agent iterative feedbacks to improve alert rule quality. Evaluated on four real-world datasets from Company-X's services, AlertGuardian significantly mitigates alert fatigue (94.8\% alert reduction ratios) and accelerates fault diagnosis (90.5\% diagnosis accuracy). Moreover, AlertGuardian improves 1,174 alert rules, with 375 accepted by SREs (32% acceptance rate). Finally, we share success stories and lessons learned about alert life-cycle management after the deployment of AlertGuardian in Company-X.

AlertGuardian: Intelligent Alert Life-Cycle Management for Large-scale Cloud Systems

TL;DR

The paper tackles alert fatigue in large-scale cloud systems by introducing AlertGuardian, a lifecycle-aware framework that combines a lightweight graph-based denoising module with Retrieval-Augmented Generation (RAG) powered alert summarization and an offline, multi-agent rule refinement workflow. The approach addresses three core gaps: online noise filtering, contextual alert summaries, and iterative rule improvement, validated on four real-world datasets with substantial reductions in alert volume and high-quality diagnostic narratives. Results show roughly 94–95% alert reduction, 98.5% action accuracy in summaries, and 31–33% rule acceptance across datasets, underscoring practical impact on MTTR and operator workload. The work demonstrates how a hybrid of efficient graph models and LLMs, reinforced by RAG and human-in-the-loop validation, can sustainably improve alert management in dynamic cloud environments, with concrete deployment lessons and guidelines for future research.

Abstract

Alerts are critical for detecting anomalies in large-scale cloud systems, ensuring reliability and user experience. However, current systems generate overwhelming volumes of alerts, degrading operational efficiency due to ineffective alert life-cycle management. This paper details the efforts of Company-X to optimize alert life-cycle management, addressing alert fatigue in cloud systems. We propose AlertGuardian, a framework collaborating large language models (LLMs) and lightweight graph models to optimize the alert life-cycle through three phases: Alert Denoise uses graph learning model with virtual noise to filter noise, Alert Summary employs Retrieval Augmented Generation (RAG) with LLMs to create actionable summary, and Alert Rule Refinement leverages multi-agent iterative feedbacks to improve alert rule quality. Evaluated on four real-world datasets from Company-X's services, AlertGuardian significantly mitigates alert fatigue (94.8\% alert reduction ratios) and accelerates fault diagnosis (90.5\% diagnosis accuracy). Moreover, AlertGuardian improves 1,174 alert rules, with 375 accepted by SREs (32% acceptance rate). Finally, we share success stories and lessons learned about alert life-cycle management after the deployment of AlertGuardian in Company-X.
Paper Structure (29 sections, 3 equations, 12 figures, 4 tables)

This paper contains 29 sections, 3 equations, 12 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Background
  4. Problems in Our Existing Alert Management System
  5. Problem 1: Frequent Alert Storms
  6. Problem 2: Low-quality Alert Rules
  7. Why Our Existing Alert Systems Do Not Help?
  8. Limitation 1: Obsolete Alert Denoising Mechanism
  9. Limitation 2: Missing Alert Summary Process
  10. Limitation 3: Missing Alert Rule Refinement Process
  11. Experience of Designing AlertGuardian
  12. Alert Denoise
  13. Alert Preprocess
  14. Graph Learning Model Training
  15. Model inference
  16. ...and 14 more sections

Figures (12)

  • Figure 1: The process of alert life-cycle management.
  • Figure 2: Examples of alert rules and alert instances.
  • Figure 3: The percentage breakdown of average per-minute alert volume in Company-X.
  • Figure 4: Change trend of alert volume under 59,607 alert rules in system A.
  • Figure 5: Change trend of alert volume under 3,544 alert rules in system B.
  • ...and 7 more figures