Semi-Supervised Learning for Anomaly Traffic Detection via Bidirectional Normalizing Flows

TL;DR

A semi-supervised anomaly detection framework that requires only two modules to detect anomalous samples, leading to a considerable reduction in model size and exhibits good generalisation performance across datasets.

Abstract

With the rapid development of the Internet, various types of anomaly traffic are threatening network security. We consider the problem of anomaly network traffic detection and propose a three-stage anomaly detection framework using only normal traffic. Our framework can generate pseudo anomaly samples without prior knowledge of anomalies to achieve the detection of anomaly data. Firstly, we employ a reconstruction method to learn the deep representation of normal samples. Secondly, these representations are normalized to a standard normal distribution using a bidirectional flow module. To simulate anomaly samples, we add noises to the normalized representations which are then passed through the generation direction of the bidirectional flow module. Finally, a simple classifier is trained to differentiate the normal samples and pseudo anomaly samples in the latent space. During inference, our framework requires only two modules to detect anomalous samples, leading to a considerable reduction in model size. According to the experiments, our method achieves the state of-the-art results on the common benchmarking datasets of anomaly network traffic detection. The code is given in the https://github.com/ZxuanDang/ATD-via-Flows.git

Figures (4)

• Figure 1: (a) Anomalies in images comprise of both colour and shape. Based on prior knowledge of anomaly patterns, images can simulate anomalies by introducing "noise" li2021cutpastezavrtanik2021draem. (b) Network traffic anomaly patterns are difficult to generalise. Simulating abnormal network traffic packets by directly introducing "noise" may destroy the semantic information of the data packets and produce meaningless pseudo anomalies, as shown in Section \ref{['ablation study']}. Our framework is able to simulate anomaly samples without prior knowledge of anomaly patterns.
• Figure 2: An overview of our framework for anomaly detection. $c$ corresponds to the representation of normal packets in the standard normal space, and $\eta$ corresponds to the noise vector sampled from a Gaussian Distribution. Feature Extractor is trained to perform deep feature extraction on one-dimensional normal packets. Bidirectional Flow Module is trained to normalize the representation of normal packets to a standard normal distribution. During the training of Classifier, the representation of normal packets is normalized to a standard normal distribution. In the standard normal space, we introduce noise sampled from a Gaussian Distribution to the normalized representation, and then simulate the representation of anomaly traffic through the generation direction. Classifier is trained to distinguish the representation of normal packets $z$ and the simulated representation of anomaly packets $\hat{z}$, enabling efficient anomaly detection. In the inference phase, our method can achieve good anomaly detection by maintaining only two modules, which greatly reduces the size of the model.
• Figure 3: Histogram with density curve. We plot the detection result histogram of the samples in the testing sets of the three datasets. The curve represents the kernel density estimation of the results. Our method is more effective in distinguishing between anomaly and normal traffic on the "UNB-CIC Tor and non-Tor" and the "ISCX VPN and non-VPN" datasets. Although the results of distinguishing normal and anomaly traffic on the "DataCon2020" dataset are not satisfactory, they are still better than those achieved by other methods.
• Figure 4: T-SNE visualization of representations in latent space. We plot the features of the normal, anomaly, and synthetic anomaly samples. It can be seen that our synthetic anomaly samples do not overlap well with real anomaly samples, but they are significantly different from normal samples. The model learns how to accurately identify normal traffic by distinguishing between them.