Nissist: An Incident Mitigation Copilot based on Troubleshooting Guides

TL;DR

The paper tackles the challenge of incident mitigation in enterprise cloud services where unstructured Troubleshooting Guides (TSGs) impede rapid resolution, especially for new on-call engineers. It proposes Nissist, a semi-automated framework that builds a structured knowledge base from TSGs and mitigation histories using Large Language Models and a multi-agent system to interpret intents, retrieve relevant knowledge nodes, and plan actions in iterative cycles with an execution engine. Key contributions include a JSON-based knowledge-node representation with explicit flow and linker fields, cross-TSG connection discovery, and a semi-automated action planner with a domain-expert post-processor to reduce hallucinations. Experimental use cases and human evaluations show substantial Time to Mitigate improvements and reduced manual intervention, indicating practical impact for improving reliability and lowering on-call fatigue.

Abstract

Effective incident management is pivotal for the smooth operation of enterprises-level cloud services. In order to expedite incident mitigation, service teams compile troubleshooting knowledge into Troubleshooting Guides (TSGs) accessible to on-call engineers (OCEs). While automated pipelines are enabled to resolve the most frequent and easy incidents, there still exist complex incidents that require OCEs' intervention. However, TSGs are often unstructured and incomplete, which requires manual interpretation by OCEs, leading to on-call fatigue and decreased productivity, especially among new-hire OCEs. In this work, we propose Nissist which leverages TSGs and incident mitigation histories to provide proactive suggestions, reducing human intervention. Leveraging Large Language Models (LLM), Nissist extracts insights from unstructured TSGs and historical incident mitigation discussions, forming a comprehensive knowledge base. Its multi-agent system design enhances proficiency in precisely discerning user queries, retrieving relevant information, and delivering systematic plans consecutively. Through our user case and experiment, we demonstrate that Nissist significant reduce Time to Mitigate (TTM) in incident mitigation, alleviating operational burdens on OCEs and improving service reliability. Our demo is available at https://aka.ms/nissist_demo.

Figures (2)

• Figure 1: The Semi-Automated Incident Mitigation Framework with Nissist. When incidents exceed automation capabilities, OCEs engage in iterative interactions with Nissist. Nissist interprets OCE intents, retrieves knowledge from the knowledge base, and formulates actions. Executable action is conducted with the execution engine, generating insights for the next round node retrieval and action planning in an automative iteration manner (purple dashed box). Actions that cannot be carried out by the execution engines are then delegated to OCEs for manual execution. The knowledge base is built offline with LLM-extracted knowledge from unstructured TSGs and mitigation history.
• Figure 2: A use case demonstrates that Nissist mitigates the connection lost incident between Service A and Service B. For simplicity, only the first three iterations are presented. $3a$ & $3b$ show two different mitigate paths due to two different execution results. In particular, $3b$ indicates that Nissist can leverage knowledge cross TSG (the blue-colored node is extracted from another TSG).