AutoMalDesc: Large-Scale Script Analysis for Cyber Threat Research

TL;DR

AutoMalDesc tackles the challenge of producing thorough natural language explanations for threat detections under limited labeled data. It introduces a self-training pipeline that starts from a seed set of 900 expert-labeled scripts and expands to over 100K samples across five scripting languages, using LLM annotators and multi-temperature pseudo-labeling with rigorous filtering. The approach yields statistically significant gains in language and malware detection, with qualitative assessments by humans and LLM judges confirming the linguistic coherence of generated summaries. The authors publish a public dataset and evaluation framework to enable reproducibility and further research in scalable, interpretable cyber threat analysis.

Abstract

Generating thorough natural language explanations for threat detections remains an open problem in cybersecurity research, despite significant advances in automated malware detection systems. In this work, we present AutoMalDesc, an automated static analysis summarization framework that, following initial training on a small set of expert-curated examples, operates independently at scale. This approach leverages an iterative self-paced learning pipeline to progressively enhance output quality through synthetic data generation and validation cycles, eliminating the need for extensive manual data annotation. Evaluation across 3,600 diverse samples in five scripting languages demonstrates statistically significant improvements between iterations, showing consistent gains in both summary quality and classification accuracy. Our comprehensive validation approach combines quantitative metrics based on established malware labels with qualitative assessment from both human experts and LLM-based judges, confirming both technical precision and linguistic coherence of generated summaries. To facilitate reproducibility and advance research in this domain, we publish our complete dataset of more than 100K script samples, including annotated seed (0.9K) and test (3.6K) datasets, along with our methodology and evaluation framework.

Figures (19)

• Figure 1: Self-training methodology. A seed dataset of 900 high-quality labeled samples initiates semi-supervised learning through pseudo-labeling. An LLM trained on seed data generates filtered pseudo-labels for unlabeled data, enabling iterative improvement with expanding training data.
• Figure 2: Evolution of model predictions for a malicious JavaScript program. The pretrained (base) model is unable to recognize the malicious intent of the code, focusing on surface-level details. Through self-learning, the model learns to detect malware and identify the attack mechanism.
• Figure 3: Representative examples from our dataset showing malicious and benign scripts. Seed and test set summaries are generated using LLMs informed by SANDBOX detonation reports, while training set summaries use our iterative methodology without detonation data. Code references are formatted in monospace for clarity.
• Figure 4: Pseudo-label quality filtering pipeline. The LLM annotator generates metadata and natural language explanations using multiple configurations (e.g. temperature settings) to ensure reliability. Samples pass filtering if metadata remains consistent across configurations and meets confidence thresholds. An LLM judge then verifies coherence between generated explanations and metadata labels.
• Figure 5: Training data retention based on confidence scores. Distribution of model confidence scores, derived from logit probabilities of maliciousness predictions. In our case, a 90% threshold balances quality and quantity, though this value is task-specific. Best viewed in color.