Streaming Large-Scale Electron Microscopy Data to a Supercomputing Facility

TL;DR

This work addresses the data deluge from the $480\,\mathrm{Gbit\,s^{-1}}$ 4D Camera by replacing disk I/O with a RAM-to-RAM streaming pipeline over a high-speed network to NERSC, enabling real-time HPC processing. It uses ZeroMQ-based sockets and a central NCEM aggregator to route frame-aligned data to NERSC compute nodes, where electron counting is performed with stempy and results stored as sparse HDF5; the workflow is integrated into the Distiller web interface for user accessibility. In a production test, over $10{,}\mathrm{TB}$ of raw data were streamed and electron-counted in real time, compressing to $125{,}\mathrm{GB}$, and the streaming workflow achieved $5$–$14\times$ higher raw throughput than the conventional file-transfer approach, with data available within about $30{,}\mathrm{s}$ of acquisition. The study demonstrates a practical, user-friendly integration of HPC resources with EOS facilities via Distiller, reducing human error and enabling more rapid, complex experiments across electron microscopy and similar data-intensive domains.

Abstract

Data management is a critical component of modern experimental workflows. As data generation rates increase, transferring data from acquisition servers to processing servers via conventional file-based methods is becoming increasingly impractical. The 4D Camera at the National Center for Electron Microscopy (NCEM) generates data at a nominal rate of 480 Gbit/s (87,000 frames/s) producing a 700 GB dataset in fifteen seconds. To address the challenges associated with storing and processing such quantities of data, we developed a streaming workflow that utilizes a high-speed network to connect the 4D Camera's data acquisition (DAQ) system to supercomputing nodes at the National Energy Research Scientific Computing Center (NERSC), bypassing intermediate file storage entirely. In this work, we demonstrate the effectiveness of our streaming pipeline in a production setting through an hour-long experiment that generated over 10 TB of raw data, yielding high-quality datasets suitable for advanced analyses. Additionally, we compare the efficacy of this streaming workflow against the conventional file-transfer workflow by conducting a post-mortem analysis on historical data from experiments performed by real users. Our findings show that the streaming workflow significantly improves data turnaround time, enables real-time decision-making, and minimizes the potential for human error by eliminating manual user interactions.

Figures (5)

• Figure 1: Schematic illustrating both the DAQ system and the initial mitigation strategies for managing large-scale 4D-STEM datasets generated at NCEM. A user begins an experiment using the TEAM 0.5 microscope software for the four-sector 4D Camera (a). The camera is connected to data receiving servers through FPGAs (b). Each server ingests all data into RAM and subsequently writes it to an eight TB flash storage system (c), which takes around 150 s for a 700 GB dataset. The data is either processed locally at NCEM on a single server with ten CPU cores (d), or transferred to NERSC's filesystems (e) and processed with more robust compute resources (f). Data processing is illustrated in panel (g), showing the assembly of disconnected sectors into coherent frames and subsequent electron counting of these frames. This processed data is saved in a single HDF5 file. The Distiller web application (h) enables the user (i) to initiate file transfers to NERSC's file systems, perform electron counting, and launch analysis notebooks in NERSC's Jupyter environment.
• Figure 2: Schematic comparison of the data streaming pipeline (blue pathway, a-b-d) with the file transfer pipeline (red pathway, a-c-d). Starting from the data receivers (a), the streaming approach employs ZeroMQ sockets to bypass raw file disk storage at NCEM, enabling direct RAM-to-RAM transfer. Sockets are created on the data receivers, a centralized aggregator server at NCEM, and NERSC compute nodes to facilitate this transmission. Conversely, the file transfer approach requires several intermediate file storage operations to move the data from NCEM to NERSC. In both pathways, the thick vertical line indicates the network border between NCEM and NERSC. Using stempy, the data is electron counted and saved in a single HDF5 file for further processing (d).
• Figure 3: Reconstructions of the same gold nanoparticle using (a) parallax and (b) ptychography over the course of the nearly hour long experiment. (i), (ii), and (iii) display reconstructions from the experiment's start, middle, and end.
• Figure 4: Fitted parameters from reconstructions in Fig. \ref{['fig:parallax_ptycho']}. (a) Defocus (C1) drift of the TEAM 0.5 during the experiment, 0.3 pm $\mathrm{s}^{-1}$. (b) Probe astigmatism (X and Y) drift throughout the experiment, both drifting at about 0.2 pm $\mathrm{s}^{-1}$. (a) and (b) were both fit using the data in Fig. \ref{['fig:parallax_ptycho']}a. (c) Lateral drift of the sample, fit with cross-correlation using the first ptychography reconstruction as the basis $\left( x = 0, \, y = 0 \right)$.
• Figure 5: Timeline diagrams of the streaming workflow compared to the file transfer workflow for two different types of experiments: (a) multi-scan, where data is automatically acquired at regular intervals similar to the stability experiment; and (b) 4D-STEM tomography, where data is collected at semi-regular intervals, but adjustments must be made to the microscope between acquisitions. The left side of each horizontal bar represents the acquisition start time, and the right side indicates the time when the electron-counted data is available at NERSC. (c) and (d) qualitatively indicate the serial steps taken within each of these bars for streaming and file transfer, respectively. The arrows in (a) represent the twenty-fourth acquisition in both workflows.