Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Low Latency, High Bandwidth Streaming of Experimental Data with EJFAT

Ilya Baldin, Michael Goodrich, Vardan Gyurjyan, Graham Heyes, Derek Howard, Yatish Kumar, David Lawrence, Brad Sawatzky, Stacey Sheldon, Carl Timmer

TL;DR

Modern DAQ systems require ultra-high-throughput, low-latency streaming across wide-area networks while preserving flexible in-network processing. EJFAT provides an FPGA-accelerated edge-to-cluster transport fabric with a NAT-like load-balancing dataplane and a programmable control plane that routes data-aggregation events to compute nodes, dynamically adjusting to workload. A CLAS12 demonstration at ESnet/NERSC achieved over 100 Gbps streaming with real-time reconstruction, validating end-to-end viability and minimizing intermediate storage along the stream. The work showcases DOE synergy through the Integrated Research Infrastructure (IRI) and points to practical impact for real-time, scalable data processing in large scientific facilities.

Abstract

Thomas Jefferson National Accelerator Facility (JLab) has partnered with Energy Sciences Network (ESnet) to define and implement an edge to compute cluster computational load balancing acceleration architecture. The ESnet-JLab FPGA Accelerated Transport (EJFAT) architecture focuses on FPGA acceleration to address compression, fragmentation, UDP packet destination redirection (Network Address Translation (NAT)) and decompression and reassembly. EJFAT seamlessly integrates edge and cluster computing to support direct processing of streamed experimental data. This will directly benefit the JLab science program as well as data centers of the future that require high throughput and low latency for both time-critical data acquisition systems and data center workflows. The EJFAT project will be presented along with how it is synergistic with other DOE activities such as an Integrated Research Infrastructure (IRI), and recent results using data sources at JLab, an EJFAT LB at ESnet, and computational cluster resources at Lawrence Berkeley National Laboratory (LBNL).

Low Latency, High Bandwidth Streaming of Experimental Data with EJFAT

TL;DR

Modern DAQ systems require ultra-high-throughput, low-latency streaming across wide-area networks while preserving flexible in-network processing. EJFAT provides an FPGA-accelerated edge-to-cluster transport fabric with a NAT-like load-balancing dataplane and a programmable control plane that routes data-aggregation events to compute nodes, dynamically adjusting to workload. A CLAS12 demonstration at ESnet/NERSC achieved over 100 Gbps streaming with real-time reconstruction, validating end-to-end viability and minimizing intermediate storage along the stream. The work showcases DOE synergy through the Integrated Research Infrastructure (IRI) and points to practical impact for real-time, scalable data processing in large scientific facilities.

Abstract

Thomas Jefferson National Accelerator Facility (JLab) has partnered with Energy Sciences Network (ESnet) to define and implement an edge to compute cluster computational load balancing acceleration architecture. The ESnet-JLab FPGA Accelerated Transport (EJFAT) architecture focuses on FPGA acceleration to address compression, fragmentation, UDP packet destination redirection (Network Address Translation (NAT)) and decompression and reassembly. EJFAT seamlessly integrates edge and cluster computing to support direct processing of streamed experimental data. This will directly benefit the JLab science program as well as data centers of the future that require high throughput and low latency for both time-critical data acquisition systems and data center workflows. The EJFAT project will be presented along with how it is synergistic with other DOE activities such as an Integrated Research Infrastructure (IRI), and recent results using data sources at JLab, an EJFAT LB at ESnet, and computational cluster resources at Lawrence Berkeley National Laboratory (LBNL).
Paper Structure (8 sections, 3 figures)

This paper contains 8 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Accelerated Load Balanced Transport Fabric
  3. Source Data Preparation
  4. LB dataplane forwarding
  5. LB Control Plane
  6. EJFAT Demonstration
  7. DOE Synergies
  8. Summary and Conclusions

Figures (3)

  • Figure 1: EJFAT End to End Packet Flow and Processing
  • Figure 2: CLAS12 Remote Data Stream Processing Data-Flow Diagram. The EJFAT packetizer segments data aggregation events into UDP packets, facilitating their transport. The EJFAT reassembly engine reconstructs these UDP packets into a cohesive data aggregation event for backend processing. The nodes in the ERSAP data pipeline represent CLAS12 reconstruction actors that process data corresponding to specific CLAS12 detector components. Blue lines are control and orange lines data paths.
  • Figure 3: These plots illustrate data throughput into and out of ESnet during the concept validation experiment, and a plot from the data processing validation highlighting the pi0 missing mass reconstruction.