RACONTEUR: A Knowledgeable, Insightful, and Portable LLM-Powered Shell Command Explainer

TL;DR

Raconteur tackles the challenge of making malicious shell commands understandable to security analysts by infusing an LLM with expert knowledge, enabling faithful behavior explanations and attacker-intent reasoning. It integrates a Behavior Explainer, an Intent Identifier grounded in MITRE ATT&CK through a BD2Vec embedding, and a Doc-Augmented Enhancer that retrieves private documentation via CD2Vec to handle unseen commands. The system is trained on a large, diverse dataset and evaluated against strong baselines, showing substantial improvements in explanation quality, intent identification, and user perception in both English and Chinese. This approach offers a portable, private-command-capable tool for shell log auditing with potential for extension to other log modalities and real-world security operations.

Abstract

Malicious shell commands are linchpins to many cyber-attacks, but may not be easy to understand by security analysts due to complicated and often disguised code structures. Advances in large language models (LLMs) have unlocked the possibility of generating understandable explanations for shell commands. However, existing general-purpose LLMs suffer from a lack of expert knowledge and a tendency to hallucinate in the task of shell command explanation. In this paper, we present Raconteur, a knowledgeable, expressive and portable shell command explainer powered by LLM. Raconteur is infused with professional knowledge to provide comprehensive explanations on shell commands, including not only what the command does (i.e., behavior) but also why the command does it (i.e., purpose). To shed light on the high-level intent of the command, we also translate the natural-language-based explanation into standard technique & tactic defined by MITRE ATT&CK, the worldwide knowledge base of cybersecurity. To enable Raconteur to explain unseen private commands, we further develop a documentation retriever to obtain relevant information from complementary documentations to assist the explanation process. We have created a large-scale dataset for training and conducted extensive experiments to evaluate the capability of Raconteur in shell command explanation. The experiments verify that Raconteur is able to provide high-quality explanations and in-depth insight of the intent of the command.

Figures (6)

• Figure 1: Inference pipeline of Raconteur. The system receives queries from analysts and IDS. First, related documentations of the requested command are retrieved. Then a documentation-augmented prompt is created and fed into the command behavior explainer for analysis. The summarized behavior of the command is utilized for intent identification.
• Figure 2: Dataset generation pipeline of Raconteur. The prompt diversification module is utilized to generate prompts with diversified expressions and questions. In the response professionalization module, commands from different data sources are prompted with corresponding meta-information and command documentations for accurate responses.
• Figure 3: Intent identification design of Raconteur. In the training phase, a Text2Vec model is fine-tuned to acquire an enhanced BD2Vec model, which maps the behavior description from LLMs and the standard description into the same embedding space. In the inference phase, the behavior description is encoded into a vector for comparison with a set of standard technique descriptions.
• Figure 4: The impact of temperature and top-p on the performance of Raconteur.
• Figure 5: ROC curves of five Text2Vec models. Our CD2Vec model obtains the highest AUC, indicating the best discriminating ability.