Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FitNets: An Adaptive Framework to Learn Accurate Traffic Distributions

Alexander Dietmüller, Albert Gran Alcoz, Laurent Vanbever

TL;DR

FitNets addresses the challenge of learning accurate traffic distributions for network management by bridging learning in the control plane with real-time verification in the data plane. It uses non-parametric Kernel Density Estimation to learn arbitrary distributions, and proper scoring rules, particularly the Quadratic Score, to quantify estimation accuracy; scores are normalized to enable cross-distribution comparability and to guide adaptive sampling. A feedback loop enables per-task sampling-rate adaptation to meet operator goals, achieving both resource-efficient and high-accuracy monitoring. The system is implemented in Python and P4, tested on real and synthetic traces, and shown to scale to hundreds of densities at up to tens of millions of samples per second, with reliable accuracy estimates and practical adaptation strategies for diverse traffic patterns.

Abstract

Learning precise distributions of traffic features (e.g., burst sizes, packet inter-arrival time) is still a largely unsolved problem despite being critical for management tasks such as capacity planning or anomaly detection. A key limitation nowadays is the lack of feedback between the control plane and the data plane. Programmable data planes offer the opportunity to create systems that let data- and control plane to work together, compensating their respective shortcomings. We present FitNets, an adaptive network monitoring system leveraging feedback between the data- and the control plane to learn accurate traffic distributions. In the control plane, FitNets relies on Kernel Density Estimators which allow to provably learn distributions of any shape. In the data plane, FitNets tests the accuracy of the learned distributions while dynamically adapting data collection to the observed distribution fitness, prioritizing under-fitted features. We have implemented FitNets in Python and P4 (including on commercially available programmable switches) and tested it on real and synthetic traffic traces. FitNets is practical: it is able to estimate hundreds of distributions from up to 60 millions samples per second, while providing accurate error estimates and adapting to complex traffic patterns.

FitNets: An Adaptive Framework to Learn Accurate Traffic Distributions

TL;DR

FitNets addresses the challenge of learning accurate traffic distributions for network management by bridging learning in the control plane with real-time verification in the data plane. It uses non-parametric Kernel Density Estimation to learn arbitrary distributions, and proper scoring rules, particularly the Quadratic Score, to quantify estimation accuracy; scores are normalized to enable cross-distribution comparability and to guide adaptive sampling. A feedback loop enables per-task sampling-rate adaptation to meet operator goals, achieving both resource-efficient and high-accuracy monitoring. The system is implemented in Python and P4, tested on real and synthetic traces, and shown to scale to hundreds of densities at up to tens of millions of samples per second, with reliable accuracy estimates and practical adaptation strategies for diverse traffic patterns.

Abstract

Learning precise distributions of traffic features (e.g., burst sizes, packet inter-arrival time) is still a largely unsolved problem despite being critical for management tasks such as capacity planning or anomaly detection. A key limitation nowadays is the lack of feedback between the control plane and the data plane. Programmable data planes offer the opportunity to create systems that let data- and control plane to work together, compensating their respective shortcomings. We present FitNets, an adaptive network monitoring system leveraging feedback between the data- and the control plane to learn accurate traffic distributions. In the control plane, FitNets relies on Kernel Density Estimators which allow to provably learn distributions of any shape. In the data plane, FitNets tests the accuracy of the learned distributions while dynamically adapting data collection to the observed distribution fitness, prioritizing under-fitted features. We have implemented FitNets in Python and P4 (including on commercially available programmable switches) and tested it on real and synthetic traffic traces. FitNets is practical: it is able to estimate hundreds of distributions from up to 60 millions samples per second, while providing accurate error estimates and adapting to complex traffic patterns.
Paper Structure (53 sections, 13 equations, 13 figures, 1 table)

This paper contains 53 sections, 13 equations, 13 figures, 1 table.

Table of Contents

  1. Introduction
  2. FitNets
  3. Overview
  4. Example
  5. Estimation of Traffic Distributions
  6. Kernel Density Estimation
  7. Density Estimation
  8. Asymptotic Convergence
  9. Optimal Estimation
  10. Proper Scoring Rules
  11. Propriety
  12. Distance
  13. Quadratic Score
  14. Data Plane Scoring
  15. Normalization
  16. ...and 38 more sections

Figures (13)

  • Figure 1: FitNets learns from traffic to optimizes monitoring.
  • Figure 2: Kernel Density Estimation
  • Figure 3: Normalization makes performance comparable.
  • Figure 4: Monitoring tasks define where to monitor which feature under which constraints on the traffic.
  • Figure 5: In the control plane, FitNets estimates density and solves the linear optimization problem to compute their accuracy. Additionally, it optimizes the sampling rate.
  • ...and 8 more figures