Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PSSketch: Finding Persistent and Sparse Flow with High Accuracy and Efficiency

Jiayao Wang, Qilong Shi, Xiyan Liang, Han Wang, Wenjun Li, Ziling Wei, Weizhe Zhang, Shuhui Chen

TL;DR

This paper tackles detecting persistent sparse flows (PS) in network traffic by introducing an anomaly-boundary criterion based on persistence and density to separate PS flows from regular traffic. It presents PSSketch, a dual-layer sketch with a Competition Layer (CL) and a Protection Layer (PL) that protects PS flows using overflow counters, complemented by insert/update/contend/query operations and optimizations (Prune, Burst Elimination, One-time Traversal) plus SIMD to boost throughput. The authors provide a mathematical foundation for density behavior under Poisson arrivals, derive error and complexity bounds, and show that PSSketch achieves substantial memory savings and accuracy gains (notably improved F1 and ARE) while delivering higher throughput relative to state-of-the-art PS-flow methods. Experiments on CAIDA, Campus, and MAWI traces demonstrate the method’s effectiveness under varying memory and load conditions, highlighting its practicality for real-time threat detection and resource management in networks.

Abstract

Finding persistent sparse (PS) flow is critical to early warning of many threats. Previous works have predominantly focused on either heavy or persistent flows, with limited attention given to PS flows. Although some recent studies pay attention to PS flows, they struggle to establish an objective criterion due to insufficient data-driven observations, resulting in reduced accuracy. In this paper, we define a new criterion "anomaly boundary" to distinguish PS flows from regular flows. Specifically, a flow whose persistence exceeds a threshold will be protected, while a protected flow with a density lower than a threshold is reported as a PS flow. We then introduce PSSketch, a high-precision layered sketch to find PS flows. PSSketch employs variable-length bitwise counters, where the first layer tracks the frequency and persistence of all flows, and the second layer protects potential PS flows and records overflow counts from the first layer. Some optimizations have also been implemented to reduce memory consumption further and improve accuracy. The experiments show that PSSketch reduces memory consumption by an order of magnitude compared to the strawman solution combined with existing work. Compared with SOTA solutions for finding PS flows, it outperforms up to 2.94x in F1 score and reduces ARE by 1-2 orders of magnitude. Meanwhile, PSSketch achieves a higher throughput than these solutions.

PSSketch: Finding Persistent and Sparse Flow with High Accuracy and Efficiency

TL;DR

This paper tackles detecting persistent sparse flows (PS) in network traffic by introducing an anomaly-boundary criterion based on persistence and density to separate PS flows from regular traffic. It presents PSSketch, a dual-layer sketch with a Competition Layer (CL) and a Protection Layer (PL) that protects PS flows using overflow counters, complemented by insert/update/contend/query operations and optimizations (Prune, Burst Elimination, One-time Traversal) plus SIMD to boost throughput. The authors provide a mathematical foundation for density behavior under Poisson arrivals, derive error and complexity bounds, and show that PSSketch achieves substantial memory savings and accuracy gains (notably improved F1 and ARE) while delivering higher throughput relative to state-of-the-art PS-flow methods. Experiments on CAIDA, Campus, and MAWI traces demonstrate the method’s effectiveness under varying memory and load conditions, highlighting its practicality for real-time threat detection and resource management in networks.

Abstract

Finding persistent sparse (PS) flow is critical to early warning of many threats. Previous works have predominantly focused on either heavy or persistent flows, with limited attention given to PS flows. Although some recent studies pay attention to PS flows, they struggle to establish an objective criterion due to insufficient data-driven observations, resulting in reduced accuracy. In this paper, we define a new criterion "anomaly boundary" to distinguish PS flows from regular flows. Specifically, a flow whose persistence exceeds a threshold will be protected, while a protected flow with a density lower than a threshold is reported as a PS flow. We then introduce PSSketch, a high-precision layered sketch to find PS flows. PSSketch employs variable-length bitwise counters, where the first layer tracks the frequency and persistence of all flows, and the second layer protects potential PS flows and records overflow counts from the first layer. Some optimizations have also been implemented to reduce memory consumption further and improve accuracy. The experiments show that PSSketch reduces memory consumption by an order of magnitude compared to the strawman solution combined with existing work. Compared with SOTA solutions for finding PS flows, it outperforms up to 2.94x in F1 score and reduces ARE by 1-2 orders of magnitude. Meanwhile, PSSketch achieves a higher throughput than these solutions.
Paper Structure (39 sections, 14 theorems, 18 equations, 19 figures, 1 table)

This paper contains 39 sections, 14 theorems, 18 equations, 19 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Background
  4. Related Works
  5. Observation and criterion
  6. Proposed methods
  7. Data Structure
  8. Operations
  9. Insert
  10. Update
  11. Contend
  12. Query
  13. Optimization
  14. Prune
  15. Burst Elimination
  16. ...and 24 more sections

Key Result

theorem 1

The expectation and variance of frequency, persistence, and density of $e$ after the $i^{th}$ window is given by:

Figures (19)

  • Figure 1: Structure of related works.
  • Figure 2: The distribution characteristics of CAIDA.
  • Figure 3: Top-2K PS flows reported under the Campus dataset.
  • Figure 4: Structure of PSSketch.
  • Figure 5: Three extra counters for each bucket.
  • ...and 14 more figures

Theorems & Definitions (14)

  • theorem 1
  • theorem 2
  • theorem 3
  • theorem 4
  • theorem 5
  • theorem 6
  • theorem 7
  • theorem 8
  • theorem 9
  • theorem 10
  • ...and 4 more