Compact Data Structures for Network Telemetry

TL;DR

This survey examines how high-speed, programmable network devices enable in-network telemetry using compact data structures. It discusses the constraints of data-plane memory and per-packet processing, and how sketches and caches are adapted (and sometimes newly designed) to perform diverse telemetry tasks—from heavy hitters and distinct counting to entropy and multi-metric analyses—without exporting massive data. The paper covers data-plane resource allocation, distributed telemetry across paths and switches, and security considerations, highlighting open challenges and promising directions for future work. By bridging theory and practice, it emphasizes how compact data structures can empower real-time network management, security, and reliability directly in the data plane. The discussion points toward richer, auto-adaptive telemetry systems that can react to network conditions with minimal controller overhead.

Abstract

Collecting and analyzing of network traffic data (network telemetry) plays a critical role in managing modern networks. Network administrators analyze their traffic to troubleshoot performance and reliability problems, and to detect and block cyberattacks. However, conventional traffic-measurement techniques offer limited visibility into network conditions and rely on offline analysis. Fortunately, network devices -- such as switches and network interface cards -- are increasingly programmable at the packet level, enabling flexible analysis of the traffic in place, as the packets fly by. However, to operate at high speed, these devices have limited memory and computational resources, leading to trade-offs between accuracy and overhead. In response, an exciting research area emerged, bringing ideas from compact data structures and streaming algorithms to bear on important networking telemetry applications and the unique characteristics of high-speed network devices. In this paper, we review the research on compact data structures for network telemetry and discuss promising directions for future research.

Figures (6)

• Figure 1: Protocol-Independent Switch Architecture (PISA) Data Plane.
• Figure 2: Count-Min Sketch. This is a sketch with $r=3$ rows and $c=10$ columns. Upon inserting a key it will be hashed by each of the hash functions and the relevant indices will be incremented by 1.
• Figure 3: Bloom filter in the data plane. A bloom filter composed of a single array using three hash functions will be separated to three separate arrays in the data plane, and each hash function will be used to access one dedicated array.
• Figure 4: Space Saving in the data plane. The Space Saving table of size $w$ is divided into $w/d$ tables for $d=3$, with one part of the table in each stage. Items are hashed into each of the tables and thus may appear in any of $d$ tables.
• Figure 5: UnivMon sketch overview.