Using Large Language Models to Support Automation of Failure Management in CI/CD Pipelines: A Case Study in SAP HANA

TL;DR

The paper investigates automating failure management in SAP HANA's CI/CD pipeline using large language models (LLMs) to extract root causes from unstructured console logs and to propose exact remediation steps. It demonstrates that providing domain knowledge, especially historical failure records, yields high accuracy in both locating the most-downstream failed job (up to $97.4\%$) and delivering exact solutions (up to $92.1\%$) in real-world SAP HANA deployments. A two-stage, domain-knowledge–augmented approach with function-calling enables end-to-end remediation, though a hybrid non-LLM strategy can be more cost-effective for frequent failures. The study highlights practical implications for industrial CI/CD automation and suggests directions for scaling with larger datasets and richer domain knowledge.

Abstract

CI/CD pipeline failure management is time-consuming when performed manually. Automating this process is non-trivial because the information required for effective failure management is unstructured and cannot be automatically processed by traditional programs. With their ability to process unstructured data, large language models (LLMs) have shown promising results for automated failure management by previous work. Following these studies, we evaluated whether an LLM-based system could automate failure management in a CI/CD pipeline in the context of a large industrial software project, namely SAP HANA. We evaluated the ability of the LLM-based system to identify the error location and to propose exact solutions that contain no unnecessary actions. To support the LLM in generating exact solutions, we provided it with different types of domain knowledge, including pipeline information, failure management instructions, and data from historical failures. We conducted an ablation study to determine which type of domain knowledge contributed most to solution accuracy. The results show that data from historical failures contributed the most to the system's accuracy, enabling it to produce exact solutions in 92.1% of cases in our dataset. The system correctly identified the error location with 97.4% accuracy when provided with domain knowledge, compared to 84.2% accuracy without it. In conclusion, our findings indicate that LLMs, when provided with data from historical failures, represent a promising approach for automating CI/CD pipeline failure management.

Figures (12)

• Figure 1: Simplified structure of a complex CI/CD pipeline hierarchy
• Figure 2: Pipeline failure due to a failed step in a sub-pipeline
• Figure 3: Control-flow diagram of the LLM-based failure-management system.
• Figure 4: Control-flow diagram of log preprocessing.
• Figure 5: Prompt template for finding cause and solution of a failure