Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Real-Time Data Processor Framework for Data Handling and Analysis of High-Energy Instruments

A. Bulgarelli, N. Parmiggiani, L. Castaldini, R. Falco, A. Di Piano, V. Fioretti, G. Panebianco, A. Rizzo

TL;DR

The paper addresses the challenge of real-time data handling for high-throughput streams in ground and space astrophysical projects and high-energy instruments. It introduces the Real-Time Data Processor Framework (rta-dp), a ZeroMQ-based solution that supports distributed pipelines configured in JSON and implemented in Python and C++. The architecture employs a Producer-Supervisor-WorkerManager-Worker chain with distinct high- and low-priority data channels, a standardized JSON message header and flexible body, and integrated monitoring, logging, and alarms, enabling scalable, real-time analysis and data handling. The authors demonstrate practical deployments in reprocessing, archiving, and online quality systems (e.g., the ASTRI project), and outline ongoing work for gamma-ray detector data handling, highlighting the framework's flexibility, performance, and applicability to diverse real-time instrumentation workflows.

Abstract

We implemented a real-time data processor (rta-dp) framework that can be used to develop real-time analysis pipelines and data handling systems to manage high-throughput data streams with distributed applications in the context of ground and space astrophysical projects and high-energy instruments. The rta-dp is based on the ZeroMQ in-memory communication framework to receive input data, share data between distributed processes, and send or receive commands and pipeline configuration. The rta-dp framework has a flexible architecture that allows the implementation of distributed analysis systems customized to the requirements of several scenarios. The rta-dp framework also provides monitoring capabilities for the running processes and sends housekeeping, logging, alarms, and informative messages that a monitoring process can acquire. We are using the rta-dp in several contexts, such as acquiring and processing data from X-ray detectors to the data quality system of the ASTRI Project, as well as reprocessing and archiving data.

The Real-Time Data Processor Framework for Data Handling and Analysis of High-Energy Instruments

TL;DR

The paper addresses the challenge of real-time data handling for high-throughput streams in ground and space astrophysical projects and high-energy instruments. It introduces the Real-Time Data Processor Framework (rta-dp), a ZeroMQ-based solution that supports distributed pipelines configured in JSON and implemented in Python and C++. The architecture employs a Producer-Supervisor-WorkerManager-Worker chain with distinct high- and low-priority data channels, a standardized JSON message header and flexible body, and integrated monitoring, logging, and alarms, enabling scalable, real-time analysis and data handling. The authors demonstrate practical deployments in reprocessing, archiving, and online quality systems (e.g., the ASTRI project), and outline ongoing work for gamma-ray detector data handling, highlighting the framework's flexibility, performance, and applicability to diverse real-time instrumentation workflows.

Abstract

We implemented a real-time data processor (rta-dp) framework that can be used to develop real-time analysis pipelines and data handling systems to manage high-throughput data streams with distributed applications in the context of ground and space astrophysical projects and high-energy instruments. The rta-dp is based on the ZeroMQ in-memory communication framework to receive input data, share data between distributed processes, and send or receive commands and pipeline configuration. The rta-dp framework has a flexible architecture that allows the implementation of distributed analysis systems customized to the requirements of several scenarios. The rta-dp framework also provides monitoring capabilities for the running processes and sends housekeeping, logging, alarms, and informative messages that a monitoring process can acquire. We are using the rta-dp in several contexts, such as acquiring and processing data from X-ray detectors to the data quality system of the ASTRI Project, as well as reprocessing and archiving data.

Paper Structure

This paper contains 3 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Software Architecture
  3. Conclusion

Figures (2)

  • Figure 1: Software architecture. The figure shows a DataProcessor system where Producers send data via ZeroMQ PUSH/PUB channels to a Supervisor, which receives low-priority (lp) and high-priority (hp) tasks (files or single events in streaming). The Supervisor distributes tasks to WorkerManagers that manage queues and assign data to multiple Workers for processing. Workers send results back through ZeroMQ channels to another DataProcessor, completing the data flow. A Monitoring process collects monitoring information, and a Command process send data to DataProcessors
  • Figure 2: Monitoring GUI. The figure shows a monitoring layout for rta-dp with a hierarchical structure of metrics. At the center there are the worker state, surrounded by the state of the WorkerManagers, and finally the outermost layer includes the states of the Supervisors, indicating detailed monitoring at both global and worker levels.