Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dependency Aware Incident Linking in Large Cloud Systems

Supriyo Ghosh, Karish Grover, Jimmy Wong, Chetan Bansal, Rakesh Namineni, Mohit Verma, Saravan Rajmohan

TL;DR

This work introduces DiLink, a dependency-aware incident linking framework for large-scale cloud systems. By jointly modeling textual incident descriptions and service dependency graphs, and aligning their multi-modal embeddings with Orthogonal Procrustes, DiLink achieves substantial gains in linking accuracy, particularly for cross-service and cross-workload incidents. Extensive experiments on 610 Microsoft services across five workloads show DiLink variants reaching an F1-score around 0.96 and a ~14% improvement over state-of-the-art baselines, with deployment underway in production. The approach reduces manual toil for on-call engineers and accelerates incident resolution by surfacing related incidents from across the service graph. The work highlights practical considerations for graph construction, embedding alignment, and real-time deployment in enterprise IcM environments.

Abstract

Despite significant reliability efforts, large-scale cloud services inevitably experience production incidents that can significantly impact service availability and customer's satisfaction. Worse, in many cases one incident can lead to multiple downstream failures due to cascading effects that creates several related incidents across different dependent services. Often time On-call Engineers (OCEs) examine these incidents in silos that lead to significant amount of manual toil and increase the overall time-to-mitigate incidents. Therefore, developing efficient incident linking models is of paramount importance for grouping related incidents into clusters so as to quickly resolve major outages and reduce on-call fatigue. Existing incident linking methods mostly leverages textual and contextual information of incidents (e.g., title, description, severity, impacted components), thus failing to leverage the inter-dependencies between services. In this paper, we propose the dependency-aware incident linking (DiLink) framework which leverages both textual and service dependency graph information to improve the accuracy and coverage of incident links not only coming from same service, but also from different services and workloads. Furthermore, we propose a novel method to align the embeddings of multi-modal (i.e., textual and graphical) data using Orthogonal Procrustes. Extensive experimental results on real-world incidents from 5 workloads of Microsoft demonstrate that our alignment method has an F1-score of 0.96 (14% gain over current state-of-the-art methods). We are also in the process of deploying this solution across 610 services from these 5 workloads for continuously supporting OCEs improving incident management and reducing manual toil.

Dependency Aware Incident Linking in Large Cloud Systems

TL;DR

This work introduces DiLink, a dependency-aware incident linking framework for large-scale cloud systems. By jointly modeling textual incident descriptions and service dependency graphs, and aligning their multi-modal embeddings with Orthogonal Procrustes, DiLink achieves substantial gains in linking accuracy, particularly for cross-service and cross-workload incidents. Extensive experiments on 610 Microsoft services across five workloads show DiLink variants reaching an F1-score around 0.96 and a ~14% improvement over state-of-the-art baselines, with deployment underway in production. The approach reduces manual toil for on-call engineers and accelerates incident resolution by surfacing related incidents from across the service graph. The work highlights practical considerations for graph construction, embedding alignment, and real-time deployment in enterprise IcM environments.

Abstract

Despite significant reliability efforts, large-scale cloud services inevitably experience production incidents that can significantly impact service availability and customer's satisfaction. Worse, in many cases one incident can lead to multiple downstream failures due to cascading effects that creates several related incidents across different dependent services. Often time On-call Engineers (OCEs) examine these incidents in silos that lead to significant amount of manual toil and increase the overall time-to-mitigate incidents. Therefore, developing efficient incident linking models is of paramount importance for grouping related incidents into clusters so as to quickly resolve major outages and reduce on-call fatigue. Existing incident linking methods mostly leverages textual and contextual information of incidents (e.g., title, description, severity, impacted components), thus failing to leverage the inter-dependencies between services. In this paper, we propose the dependency-aware incident linking (DiLink) framework which leverages both textual and service dependency graph information to improve the accuracy and coverage of incident links not only coming from same service, but also from different services and workloads. Furthermore, we propose a novel method to align the embeddings of multi-modal (i.e., textual and graphical) data using Orthogonal Procrustes. Extensive experimental results on real-world incidents from 5 workloads of Microsoft demonstrate that our alignment method has an F1-score of 0.96 (14% gain over current state-of-the-art methods). We are also in the process of deploying this solution across 610 services from these 5 workloads for continuously supporting OCEs improving incident management and reducing manual toil.
Paper Structure (49 sections, 2 equations, 7 figures, 2 tables)

This paper contains 49 sections, 2 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Incident management
  4. Incidents considered in our study
  5. Incident Linking
  6. Duplicate links.
  7. Related links.
  8. Responsible links.
  9. Cross team incident links.
  10. Methodology
  11. Overview of architecture
  12. Modeling textual information
  13. Modeling Dependency graph information
  14. Dependency graph construction
  15. Encoding dependency graph
  16. ...and 34 more sections

Figures (7)

  • Figure 1: Incident management lifecycle.
  • Figure 2: Distribution of different severity incidents.
  • Figure 3: Time to mitigate (TTM) for linked vs. all incidents (Y-axis shows the normalized TTM value with the median of TTM of all incidents as 1.).
  • Figure 4: Overall architecture of DiLink.
  • Figure 5: Performance with varying (a) number of edges in dependency graph; (b) graph embedding sizes; and (c) number of neighbourhood hops.
  • ...and 2 more figures