FaultProfIT: Hierarchical Fault Profiling of Incident Tickets in Large-scale Cloud Systems

TL;DR

FaultProfIT tackles the bottleneck of manual fault pattern profiling in large-scale cloud systems by introducing a hierarchy-aware, graph-structured approach to incident-ticket understanding. It combines a MacBERT-based incident encoder with a Graphormer-based hierarchy encoder and trains them under hierarchy-guided contrastive learning, producing robust hierarchy-aware representations for multi-label fault-pattern profiling. Evaluations on CloudA production incidents show FaultProfIT achieving a high F1-score (78.3%) and outperforming several baselines, with ablations confirming the importance of hierarchical encoding and contrastive learning. The solution is deployed in CloudA for six months, profiling over 10,000 incidents across 30+ services, enabling timely detection of fault trends and informing reliability improvements.

Abstract

Postmortem analysis is essential in the management of incidents within cloud systems, which provides valuable insights to improve system's reliability and robustness. At CloudA, fault pattern profiling is performed during the postmortem phase, which involves the classification of incidents' faults into unique categories, referred to as fault pattern. By aggregating and analyzing these fault patterns, engineers can discern common faults, vulnerable components and emerging fault trends. However, this process is currently conducted by manual labeling, which has inherent drawbacks. On the one hand, the sheer volume of incidents means only the most severe ones are analyzed, causing a skewed overview of fault patterns. On the other hand, the complexity of the task demands extensive domain knowledge, which leads to errors and inconsistencies. To address these limitations, we propose an automated approach, named FaultProfIT, for Fault pattern Profiling of Incident Tickets. It leverages hierarchy-guided contrastive learning to train a hierarchy-aware incident encoder and predicts fault patterns with enhanced incident representations. We evaluate FaultProfIT using the production incidents from CloudA. The results demonstrate that FaultProfIT outperforms state-of-the-art methods. Our ablation study and analysis also verify the effectiveness of hierarchy-guided contrastive learning. Additionally, we have deployed FaultProfIT at CloudA for six months. To date, FaultProfIT has analyzed 10,000+ incidents from 30+ cloud services, successfully revealing several fault trends that have informed system improvements.

Figures (9)

• Figure 1: An example of an incident ticket.
• Figure 2: Fault pattern example in the hierarchical taxonomy.
• Figure 3: The incident management process.
• Figure 4: Fault pattern distribution of incidents with different severities that have undergone postmortem.
• Figure 5: The overview of automatic fault pattern profiling.