RulePilot: An LLM-Powered Agent for Security Rule Generation

TL;DR

RulePilot introduces an LLM-based agent for automated SIEM rule creation and cross-platform rule conversion, bridging the gap between natural-language requirements and SIEM-specific rule grammars. It integrates a novel intermediate representation (IR), Chain-of-Thought prompting, and a reflection-based iterative optimization workflow, enabling Splunk SPL rule generation from annotations and subsequent conversion to Microsoft KQL. The approach achieves substantial gains in textual fidelity (up to 107.4% over baselines) and solid execution accuracy in real-world Splunk tests, with a case study showing time savings for junior analysts. This work advances practical, end-to-end automation for security rule management and provides open datasets and code to encourage adoption and further research.

Abstract

The real-time demand for system security leads to the detection rules becoming an integral part of the intrusion detection life-cycle. Rule-based detection often identifies malicious logs based on the predefined grammar logic, requiring experts with deep domain knowledge for rule generation. Therefore, automation of rule generation can result in significant time savings and ease the burden of rule-related tasks on security engineers. In this paper, we propose RulePilot, which mimics human expertise via LLM-based agent for addressing rule-related challenges like rule creation or conversion. Using RulePilot, the security analysts do not need to write down the rules following the grammar, instead, they can just provide the annotations such as the natural-language-based descriptions of a rule, our RulePilot can automatically generate the detection rules without more intervention. RulePilot is equipped with the intermediate representation (IR), which abstracts the complexity of config rules into structured, standardized formats, allowing LLMs to focus on generation rules in a more manageable and consistent way. We present a comprehensive evaluation of RulePilot in terms of textual similarity and execution success abilities, showcasing RulePilot can generate high-fidelity rules, outperforming the baseline models by up to 107.4% in textual similarity to ground truths and achieving better detection accuracy in real-world execution tests. We perform a case study from our industry collaborators in Singapore, showcasing that RulePilot significantly help junior analysts/general users in the rule creation process.

Figures (8)

• Figure 1: Motivation scenario SOC-motivating-examplekersten2025test:motivating-example: By combining with RulePilot, junior analysts can generate concise detection rules and conversions, significantly reducing the workload of senior experts.
• Figure 2: Overview of RulePilot, where RulePilot incorporates the Chain-of-Thought (CoT), Intermediate Representation (IR), and Reflection components. Given the NLP-based input requirements, RulePilot generates SIEM-specific rules that can be directly executed in the SIEM to detect malicious logs and trigger alerts (i.e., the malicious log).
• Figure 3: Preference inter-rater alignment between the LLM and the human evaluator. R refers to RulePilot, and B refers to Baseline. The “$>$” indicates that the method was rated higher in a pairwise comparison. The diagonal entries indicate agreement between the LLM and the human evaluator.
• Figure 4: Expert-reviewed comparison between rules generated by RulePilotversus those generated by standalone GPT-4o.
• Figure 5: Semantic-level evaluation. Radar chart of LLM-based evaluator: the inner shaded area represents the baseline model's score, while the outer contour represents RulePilot's score.