Synthesizing Diverse Network Flow Datasets with Scalable Dynamic Multigraph Generation

TL;DR

The paper tackles the challenge of obtaining real netflow graphs by introducing a scalable pipeline for generating high-fidelity synthetic dynamic multigraphs that mirror a reference network. It combines structure generation with a stochastic Kronecker graph, feature generation with CTGAN, and graph alignment with XGBoost to overlay features onto topology, enabling flexible control over node and edge counts. New metrics for accuracy and diversity are proposed, and the approach is validated on two real-world datasets (THOR and FMX), demonstrating improvements in fidelity while maintaining efficiency and scalability. The work provides a practical tool for cybersecurity research and offers a framework adaptable to other graph-based domains, with a clear exploration of the accuracy-diversity trade-off and avenues for future enhancement.

Abstract

Obtaining real-world network datasets is often challenging because of privacy, security, and computational constraints. In the absence of such datasets, graph generative models become essential tools for creating synthetic datasets. In this paper, we introduce a novel machine learning model for generating high-fidelity synthetic network flow datasets that are representative of real-world networks. Our approach involves the generation of dynamic multigraphs using a stochastic Kronecker graph generator for structure generation and a tabular generative adversarial network for feature generation. We further employ an XGBoost (eXtreme Gradient Boosting) model for graph alignment, ensuring accurate overlay of features onto the generated graph structure. We evaluate our model using new metrics that assess both the accuracy and diversity of the synthetic graphs. Our results demonstrate improvements in accuracy over previous large-scale graph generation methods while maintaining similar efficiency. We also explore the trade-off between accuracy and diversity in synthetic graph dataset creation, a topic not extensively covered in related works. Our contributions include the synthesis and evaluation of large real-world netflow datasets and the definition of new metrics for evaluating synthetic graph generative models.

Figures (6)

• Figure 1: Overview of the model approach for generating synthetic netflow graphs. The process is divided into three stages: structure generation using a stochastic Kronecker graph, feature generation with CTGAN, and graph alignment via XGBoost. This modular framework enables scalable and efficient synthesis of diverse and realistic network datasets.
• Figure 2: Conceptual illustration of the accuracy-diversity trade-off using two-dimensional Gaussian distributions. The figure contrasts two models: Model 1 (blue contours) with higher bias and lower diversity (higher variability), and Model 2 (green contours) with lower bias and higher diversity (lower variability). The red point represents the true value, highlighting the balance between achieving accuracy and maintaining diversity in model predictions.
• Figure 3: CDF plots of netflow feature distributions for the THOR dataset. These compare the alignment of start-time, duration, and port-protocol features with the reference data.
• Figure 4: CDF plots of netflow feature distributions for the FMX dataset. These compare the alignment of start-time, duration, and port-protocol features with the reference data.
• Figure 5: Visualization of the FMX reference netflow graph. This graph represents the real-world network structure used as a benchmark for evaluating the accuracy and diversity of synthetic graph generation models. Nodes and edges illustrate the connectivity and flow patterns within the network.