From Flows to Words: Can Zero-/Few-Shot LLMs Detect Network Intrusions? A Grammar-Constrained, Calibrated Evaluation on UNSW-NB15

TL;DR

The paper tackles intrusion detection under high-volume, encrypted traffic by converting network flows into concise text, augmented with interpretable flags, and enforcing grammar-bound JSON outputs. It investigates whether instruction-tuned LLMs can detect intrusions in a zero/few-shot, prompt-only regime without gradient training, using a single calibrated threshold to stabilize decisions. Across experiments on UNSW-NB15, well-tuned prompt designs with flags and calibration yield competitive macro-F1 on small, balanced sets and offer readable artifacts, though traditional tabular baselines remain more stable and faster at scale. The work provides a reproducible, auditable LLM-based detection protocol and suggests hybrid designs that combine fast tabular detectors with language-based reasoning for dynamic security environments.

Abstract

Large Language Models (LLMs) can reason over natural-language inputs, but their role in intrusion detection without fine-tuning remains uncertain. This study evaluates a prompt-only approach on UNSW-NB15 by converting each network flow to a compact textual record and augmenting it with lightweight, domain-inspired boolean flags (asymmetry, burst rate, TTL irregularities, timer anomalies, rare service/state, short bursts). To reduce output drift and support measurement, the model is constrained to produce structured, grammar-valid responses, and a single decision threshold is calibrated on a small development split. We compare zero-shot, instruction-guided, and few-shot prompting to strong tabular and neural baselines under identical splits, reporting accuracy, precision, recall, F1, and macro scores. Empirically, unguided prompting is unreliable, while instructions plus flags substantially improve detection quality; adding calibrated scoring further stabilizes results. On a balanced subset of two hundred flows, a 7B instruction-tuned model with flags reaches macro-F1 near 0.78; a lighter 3B model with few-shot cues and calibration attains F1 near 0.68 on one thousand examples. As the evaluation set grows to two thousand flows, decision quality decreases, revealing sensitivity to coverage and prompting. Tabular baselines remain more stable and faster, yet the prompt-only pipeline requires no gradient training, produces readable artifacts, and adapts easily through instructions and flags. Contributions include a flow-to-text protocol with interpretable cues, a calibration method for thresholding, a systematic baseline comparison, and a reproducibility bundle with prompts, grammar, metrics, and figures.