Data acquisition from high-rate detectors at MAX IV

TL;DR

This paper presents a scalable data acquisition architecture for MAX IV's high-frame-rate detectors, integrating detector-specific DCUs with a central Kubernetes-managed DAQ cluster and GPFS storage to enable live feedback and on-the-fly processing. It introduces the STINS streaming protocol over ZeroMQ, a Python-based stream-receiver pipeline with modular processing and a Tango-controlled detector interface, and demonstrates through end-to-end tests that the system can sustain up to $2{,}000$ fps for typical $200{,}kB$ frames and up to $4{,}GB/s$ for large frames, with performance limited by software I/O rather than network bandwidth. The work details the IT and Kubernetes infrastructure, including SR-IOV networking, GitOps deployment, and policy-driven infrastructure separation, and identifies bottlenecks in HDF5 writing while outlining clear paths for optimization and expansion to multimodal online pipelines. The practical impact lies in enabling reliable, real-time data processing and visualization for high-rate X-ray experiments, with a scalable framework ready to accommodate future detector generations and increasing data deluge.

Abstract

At MAX IV pixelated area detectors are operated at high frame rates to take advantage of the X-ray beam properties available from the fourth generation synchrotron in scattering, diffraction and imaging applications. A variety of photon counting and charge integrating detectors and sCMOS cameras have been integrated into a common data acquisition (DAQ) system in which data are streamed to a central Kubernetes cluster, mounting an IBM Storage Scale (GPFS) file system. The DAQ system provides live feedback from the detectors/cameras and extends to enable on-the-fly data processing. Control system integration via Tango provides a standardised single interface for controlling all DAQ components. Starting from an overview of the detector types in use, we describe the design and implementation of the MAX~IV detector-DAQ system and report a quantitative study of its performance in terms of data throughput and detector operating rates.

Figures (8)

• Figure 1: Overview of the MAX IV detector-DAQ scheme. A detector-specific data streamer application runs on the Detector Control Unit (DCU) which has a 40 Gb/s fast data connection (purple) to the Kubernetes DAQ cluster and a 10 Gb/s slow control connection (blue) to the beamline control network. A general purpose stream-receiver application runs on the DAQ cluster, with optional additional processing steps. Data are streamed between each layer over ZeroMQ. A further slow connection between the DAQ cluster and the beamline control network allows control and monitoring of the stream-receiver and for accessing raw or processed images at low frame rates for live-viewing. Within the DAQ cluster there is a 100 Gb/s Ethernet network for intra-node traffic separated from the data ingest. I/O to the GPFS storage happens over dedicated Infiniband fabric (green).
• Figure 2: Overview of the MAX IV IT infrastructure for DAQ with a focus on the internals of the DAQ Kubernetes cluster. The DAQ Worker Nodes, Control Plane, Load Balancers and Storage Layer distributed over two server halls (Kirk and Picard) are interconnected via 100 Gb/s Ethernet and HDR Infiniband to receive, process and store data received via ZeroMQ over dedicated 100 Gb/s ingest Ethernet.
• Figure 3: Detail of the stream-receiver application which is deployed in the DAQ cluster as a Kubernetes pod for each detector. On the data handling side the ZeroMQ PULL sockets can be parallelised over multiple workers. The collector orders the frames before they are written to file or forwarded. On the API side an HTTP interface allows monitoring of the receiver status (e.g. by the Tango device for the corresponding detector) and access to the last frame (e.g. for the live-viewer)
• Figure 4: Overview of the stream-receiver deployment with a focus on the Kubernetes resources created in the DAQ cluster by the daq-deployer Helm Chart. The stream-receiver application packaged into a container image runs in a pod with a dedicated network interface to the particular detector or camera (purple). Stream re-publishing allows the pipeline to be complemented with optional processing and/or additional beamline-specific live-view steps (in addition to the raw-image live-view always available from the stream-receiver). The pipeline control pod provides the HTTP API to manage other pipeline pods. Ingress defines HTTP URLs accessible via the Load Balancer from the control system networks (blue) to allow API and additional custom live-view access from the beamline systems.
• Figure 5: Measured operating frame rate of a stream-receiver instance deployed in the MAX IV DAQ cluster, including writing of data to disk, as a function of generated input frame rate, for different frame sizes. The linear dotted blue line shows the hypothetical performance where the stream-receiver keeps up with any generated frame rate. The Hamamatsu Orca Lightning (12M), Dectris Pilatus (2M), XSpectrum Lambda (3M) detectors have been marked on this line at their respective operating frame rates to provide perspective on the current uses of the MAX IV system.