SBASH: a Framework for Designing and Evaluating RAG vs. Prompt-Tuned LLM Honeypots

TL;DR

SBASH tackles the challenge of building realistic, secure shell-based honeypots by integrating local lightweight LLMs with optional Retrieval Augmented Generation to balance realism, accuracy, latency, and data protection. The framework decomposes into configurable components (system declaration, knowledge base, filesystem layout, and tuned LLM prompting) and employs a multi-stage processing pipeline to sanitize inputs, classify commands, and leverage RAG for context-aware responses. A proof-of-concept HoneywareX demonstrates the approach across multiple model sizes and prompt configurations, evaluated with Levenshtein distance, SBert, and BertScore, plus human realism assessments and latency/cost metrics. Key findings reveal that RAG enhances accuracy for untuned models, while system-prompt tuning can match or exceed RAG performance with lower latency, indicating a practical sweet spot in model size for on-premise honeypots. This work provides the first multi-criteria evaluation of RAG-based LLMs in shell honeypots and offers actionable guidance for secure, realistic threat intelligence gathering on local infrastructure.

Abstract

Honeypots are decoy systems used for gathering valuable threat intelligence or diverting attackers away from production systems. Maximising attacker engagement is essential to their utility. However research has highlighted that context-awareness, such as the ability to respond to new attack types, systems and attacker agents, is necessary to increase engagement. Large Language Models (LLMs) have been shown as one approach to increase context awareness but suffer from several challenges including accuracy and timeliness of response time, high operational costs and data-protection issues due to cloud deployment. We propose the System-Based Attention Shell Honeypot (SBASH) framework which manages data-protection issues through the use of lightweight local LLMs. We investigate the use of Retrieval Augmented Generation (RAG) supported LLMs and non-RAG LLMs for Linux shell commands and evaluate them using several different metrics such as response time differences, realism from human testers, and similarity to a real system calculated with Levenshtein distance, SBert, and BertScore. We show that RAG improves accuracy for untuned models while models that have been tuned via a system prompt that tells the LLM to respond like a Linux system achieve without RAG a similar accuracy as untuned with RAG, while having a slightly lower latency.

Figures (3)

• Figure 1: SBASH Framework High-level Architecture
• Figure 2: Model Performance - Execution Time of evaluated models and model sizes with RAG and without RAG
• Figure 3: Realism rating from human testers: 1=very unrealistic, 5=very realistic