Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Empowering AIOps: Leveraging Large Language Models for IT Operations Management

Arthur Vitui, Tse-Hsun Chen

TL;DR

The paper investigates empowering IT Operations Management (ITOM) with Large Language Models (LLMs) by deploying tool-enabled LLM agents in a Kubernetes/OpenShift environment and integrating them with predictive ML for capacity planning. It analyzes how different LLMs perform across accuracy, latency, and verbosity on 25 ITOM tasks, using a Retrieval Augmented Generation (RAG) setup and Kubernetes APIs, including Prometheus data. The findings show GPT-4 family delivering the best overall performance, especially for complex workflows, while OpenAI models remain most efficient in token usage; Anthropic models tend to be more verbose, and some open models exhibit reliability issues like hallucination. The work provides practical guidance on model/tool selection, discusses memory usage implications, and outlines future directions toward local deployment, alternative frameworks, and broader platform applicability to enhance the cost-efficiency and scalability of agentic ITOM solutions.

Abstract

The integration of Artificial Intelligence (AI) into IT Operations Management (ITOM), commonly referred to as AIOps, offers substantial potential for automating workflows, enhancing efficiency, and supporting informed decision-making. However, implementing AI within IT operations is not without its challenges, including issues related to data quality, the complexity of IT environments, and skill gaps within teams. The advent of Large Language Models (LLMs) presents an opportunity to address some of these challenges, particularly through their advanced natural language understanding capabilities. These features enable organizations to process and analyze vast amounts of unstructured data, such as system logs, incident reports, and technical documentation. This ability aligns with the motivation behind our research, where we aim to integrate traditional predictive machine learning models with generative AI technologies like LLMs. By combining these approaches, we propose innovative methods to tackle persistent challenges in AIOps and enhance the capabilities of IT operations management.

Empowering AIOps: Leveraging Large Language Models for IT Operations Management

TL;DR

The paper investigates empowering IT Operations Management (ITOM) with Large Language Models (LLMs) by deploying tool-enabled LLM agents in a Kubernetes/OpenShift environment and integrating them with predictive ML for capacity planning. It analyzes how different LLMs perform across accuracy, latency, and verbosity on 25 ITOM tasks, using a Retrieval Augmented Generation (RAG) setup and Kubernetes APIs, including Prometheus data. The findings show GPT-4 family delivering the best overall performance, especially for complex workflows, while OpenAI models remain most efficient in token usage; Anthropic models tend to be more verbose, and some open models exhibit reliability issues like hallucination. The work provides practical guidance on model/tool selection, discusses memory usage implications, and outlines future directions toward local deployment, alternative frameworks, and broader platform applicability to enhance the cost-efficiency and scalability of agentic ITOM solutions.

Abstract

The integration of Artificial Intelligence (AI) into IT Operations Management (ITOM), commonly referred to as AIOps, offers substantial potential for automating workflows, enhancing efficiency, and supporting informed decision-making. However, implementing AI within IT operations is not without its challenges, including issues related to data quality, the complexity of IT environments, and skill gaps within teams. The advent of Large Language Models (LLMs) presents an opportunity to address some of these challenges, particularly through their advanced natural language understanding capabilities. These features enable organizations to process and analyze vast amounts of unstructured data, such as system logs, incident reports, and technical documentation. This ability aligns with the motivation behind our research, where we aim to integrate traditional predictive machine learning models with generative AI technologies like LLMs. By combining these approaches, we propose innovative methods to tackle persistent challenges in AIOps and enhance the capabilities of IT operations management.
Paper Structure (15 sections, 4 tables)

This paper contains 15 sections, 4 tables.

Table of Contents

  1. Introduction
  2. Studied Systems
  3. Studied Open Source System
  4. Evaluation Metrics for Large Language Models
  5. Case Study Results
  6. RQ1: How accurately do different LLMs perform on a set of IT Operations tasks?
  7. RQ2: How fast do different LLMs perform on a set of IT Operations tasks?
  8. RQ3: How verbose do different LLMs perform on a set of IT Operations tasks?
  9. Discussions
  10. Understanding AI Assistants Behavior With and Without A Memory Component
  11. Understanding Model Evaluations In Context
  12. Understanding Model Strengths and Weaknesses for Advanced Reasoning Tasks
  13. Threats to Validity
  14. Related Work
  15. Conclusions