PromAssistant: Leveraging Large Language Models for Text-to-PromQL

TL;DR

PromAssistant presents a knowledge-graph–augmented, LLM-driven pipeline for converting natural language queries into PromQL. By constructing a comprehensive system-context KG from Prometheus, Kubernetes, traces, and documentation, and by integrating multi-hop retrieval with CoT prompting, it achieves accurate PromQL generation and strong retrieval accuracy on a 280-question benchmark. The study demonstrates that both metric knowledge and system component knowledge are essential for correct query generation, achieving up to 69.1% QueryAcc with GPT-4-Turbo and highlighting the approach’s scalability and potential applicability to other metric systems. Overall, the work introduces a novel DSL-generation framework for metric querying and provides a path toward more accessible, on-demand metrics analysis in complex online service systems.

Abstract

With the increasing complexity of modern online service systems, understanding the state and behavior of the systems is essential for ensuring their reliability and stability. Therefore, metric monitoring systems are widely used and become an important infrastructure in online service systems. Engineers usually interact with metrics data by manually writing domain-specific language (DSL) queries to achieve various analysis objectives. However, writing these queries can be challenging and time-consuming, as it requires engineers to have high programming skills and understand the context of the system. In this paper, we focus on PromQL, which is the metric query DSL provided by the widely used metric monitoring system Prometheus. We aim to simplify metrics querying by enabling engineers to interact with metrics data in Prometheus through natural language, and we call this task text-to-PromQL. Building upon the insight, this paper proposes PromAssistant, a Large Language Model-based text-to-PromQL framework. PromAssistant first uses a knowledge graph to describe the complex context of an online service system. Then, through the synergistic reasoning of LLMs and the knowledge graph, PromAssistant transforms engineers' natural language questions into PromQL queries. To evaluate PromAssistant, we manually construct the first text-to-PromQL benchmark dataset which contains 280 metric query questions. The experiment results show that PromAssistant is effective in text-to-PromQL and outperforms baseline approaches. To the best of our knowledge, this paper is the first study of text-to-PromQL, and PromAssistant pioneered the DSL generation framework for metric querying and analysis.

Figures (7)

• Figure 1: An Example of the Metric Data in Prometheus
• Figure 2: A Motivation Example
• Figure 3: Overview of PromAssistant
• Figure 4: Schema of the System Context Knowledge Graph
• Figure 5: Example of the Prompt and Result of Question Parsing