Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Handling Large-Scale Network Flow Records: A Comparative Study on Lossy Compression

Gabriele Merlach, Damiano Ravalico, Martino Trevisan, Fabio Palmese, Giovanni Baccichet, Alessandro E. C. Redondi

TL;DR

This paper tackles the problem of storing large-scale network flow records by evaluating lossy compression techniques to preserve downstream analytical utility. It compares Scalar Quantization, PCA, and Vector Quantization on a real campus dataset for a domain-classification task, finding that scalar quantization offers the best balance between compression and predictive accuracy, while PCA increases data entropy and reduces gzip effectiveness, and vector quantization struggles with scalability. The results provide practical guidance for reducing storage in large-scale monitoring while maintaining useful analytics. The work highlights that simple, scalable quantization methods can outperform more sophisticated dimensionality reduction or clustering approaches in this setting, with future exploration of neural autoencoder-based methods.

Abstract

Flow records, that summarize the characteristics of traffic flows, represent a practical and powerful way to monitor a network. While they already offer significant compression compared to full packet captures, their sheer volume remains daunting, especially for large Internet Service Providers (ISPs). In this paper, we investigate several lossy compression techniques to further reduce storage requirements while preserving the utility of flow records for key tasks, such as predicting the domain name of contacted servers. Our study evaluates scalar quantization, Principal Component Analysis (PCA), and vector quantization, applied to a real-world dataset from an operational campus network. Results reveal that scalar quantization provides the best tradeoff between compression and accuracy. PCA can preserve predictive accuracy but hampers subsequent entropic compression, and while vector quantization shows promise, it struggles with scalability due to the high-dimensional nature of the data. These findings result in practical strategies for optimizing flow record storage in large-scale monitoring scenarios.

Handling Large-Scale Network Flow Records: A Comparative Study on Lossy Compression

TL;DR

This paper tackles the problem of storing large-scale network flow records by evaluating lossy compression techniques to preserve downstream analytical utility. It compares Scalar Quantization, PCA, and Vector Quantization on a real campus dataset for a domain-classification task, finding that scalar quantization offers the best balance between compression and predictive accuracy, while PCA increases data entropy and reduces gzip effectiveness, and vector quantization struggles with scalability. The results provide practical guidance for reducing storage in large-scale monitoring while maintaining useful analytics. The work highlights that simple, scalable quantization methods can outperform more sophisticated dimensionality reduction or clustering approaches in this setting, with future exploration of neural autoencoder-based methods.

Abstract

Flow records, that summarize the characteristics of traffic flows, represent a practical and powerful way to monitor a network. While they already offer significant compression compared to full packet captures, their sheer volume remains daunting, especially for large Internet Service Providers (ISPs). In this paper, we investigate several lossy compression techniques to further reduce storage requirements while preserving the utility of flow records for key tasks, such as predicting the domain name of contacted servers. Our study evaluates scalar quantization, Principal Component Analysis (PCA), and vector quantization, applied to a real-world dataset from an operational campus network. Results reveal that scalar quantization provides the best tradeoff between compression and accuracy. PCA can preserve predictive accuracy but hampers subsequent entropic compression, and while vector quantization shows promise, it struggles with scalability due to the high-dimensional nature of the data. These findings result in practical strategies for optimizing flow record storage in large-scale monitoring scenarios.
Paper Structure (14 sections, 3 equations, 4 figures, 1 table)

This paper contains 14 sections, 3 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Scalar Quantization
  4. Principal Component Analysis
  5. Vector Quantization
  6. Evaluation Methodology
  7. Use Case and Dataset
  8. Dataset
  9. Use Case: Domain Classification
  10. Results
  11. Scalar Quantization
  12. Principal Component Analysis
  13. Vector Quantization
  14. Conclusion and Future Work

Figures (4)

  • Figure 1: Diagram of encoding-decoding process.
  • Figure 2: Compression ratio and F1-Score with Scalar Quantization with different compression depth $B$.
  • Figure 3: Compression ratio and Classification Performance with PCA and Scalar Quantization.
  • Figure 4: Comparative Analysis of Vector Quantization Through K-means