Reproducibility in Event-Log Research: A Parametrised Generator and Benchmark for Event-based Signatures

TL;DR

The paper tackles reproducibility challenges in event-log research by introducing a parametrised synthetic event-log generator that embeds ground-truth event-based signatures into logs. It demonstrates the utility of the generator through a large-scale benchmarking study using DBSCAN, achieving ARI values above $0.95$ on most datasets. Key contributions include the novel generator with seven controllable parameters, ground-truth annotations for precise evaluation, and a comprehensive analysis of how dataset characteristics affect clustering performance, all enabling rigorous, comparable evaluation of signature-detection techniques. The work significantly improves the ability to develop and benchmark cybersecurity analytics by providing shareable, ground-truth datasets and a clear benchmarking framework.

Abstract

Event-based datasets are crucial for cybersecurity analysis. A key use case is detecting event-based signatures, which represent attacks spanning multiple events and can only be understood once the relevant events are identified and linked. Analysing event datasets is essential for monitoring system security, but their growing volume and frequency create significant scalability and processing difficulties. Researchers rely on these datasets to develop and test techniques for automatically identifying signatures. However, because real datasets are security-sensitive and rarely shared, it becomes difficult to perform meaningful comparative evaluation between different approaches. This work addresses this evaluation limitation by offering a systematic method for generating event logs with known ground truth, enabling reproducible and comparable research. We present a novel parametrised generation technique capable of producing synthetic event datasets that contain event-based signatures for discovery. To demonstrate the capabilities of the technique, we provide a benchmark in signature detection. Our benchmarking demonstrated the suitability of DBSCAN, achieving a score greater than 0.95 Adjusted Rand Index on most generated datasets. This work enhances the ability of researchers to develop and benchmark new cybersecurity techniques, ultimately contributing to more robust and effective cybersecurity measures.

Figures (8)

• Figure 1: Example Microsoft event (ID 4670) detailing the change in security permissions
• Figure 2: Example Syslog event for an invalid user login
• Figure 3: Example of a suspicious system activity. The labels SID_S (Source Security Identifier) and SID_T (Target Security Identifier), which identify the source and target entities involved in the activity, are provided in the figure's bottom-right corner.
• Figure 4: Synthetic Event Generation Process. The red circles are used to illustrate parameter inputs to the approach, blue boxes are used to illustrate functions within the generation process, yellow boxes illustrate functional groupings, and the arrows represent the flow of information.
• Figure 5: Boxplot showing the time in milliseconds to generate the datasets. The plot focusses on the total number of events in the dataset (10K, 20K, 30K, and 40K).