An Online Data Analysis Framework for Small-Scale Physics Experiments

TL;DR

The paper introduces an online data-analysis framework tailored for small-scale physics experiments, demonstrated during HRMT-68 at CERN's HiRadMat facility. Built in Python with a JSON-configurable, modular architecture, the framework enables real-time data extraction, visualization, and feature identification across diverse diagnostics while adapting to new devices. It addresses practical challenges such as timestamp tagging and data synchronization by implementing dual central logs and leveraging CERN storage services, and it evaluates performance across multiple devices to quantify runtime and memory overhead. The work offers a reusable, adaptable solution that enhances experimental agility and fault diagnosis for collaborations with limited resources.

Abstract

A robust and flexible architecture capable of providing real-time analysis on diagnostic data is of crucial importance to physics experiments. In this paper, we present such an online framework, used in June 2025 as part of the HRMT-68 experiment, performed at the HiRadMat facility at CERN, using the Super Proton Synchrotron (SPS) beam line. HRMT-68 was a fixed-target laboratory astrophysics experiment aiming to identify plasma instabilities generated by a relativistic electron-positron beam during traversal of an argon plasma. This framework was essential for experimental data acquisition and analysis, and can be adapted for a broad range of experiments with a variety of experimental diagnostics. The framework's modular and customizable design enabled us to rapidly observe and extract emergent features from a diverse range of diagnostic data. Simultaneously, it allowed for both the introduction of new diagnostic devices and the modification of our analysis as features of interest were identified. As a result, we were able to effectively diagnose equipment malfunction, and infer the beam's response to varying bunch duration, beam intensity, and the plasma state without resorting to offline analysis, at which time adjustment or improvement would have been impossible. We present the features of this agile framework, whose codebase we have made publicly available, which can be adapted for future experiments with minimal modification.

Figures (5)

• Figure 1: Schematic diagram showing the ideal (and initially-proposed) data tagging, logging, and retrieval processes. In reality, the shot log could not be automatically generated, and the process of time-stamping data using the CERN accelerator cyclestamp proved infeasible due to instabilities in network connection.
• Figure 2: Schematic diagram showing the system of data tagging, logging, and retrieval in reality, without tagging via cyclestamp. Here, a secondary log with minimal human-readable detail is introduced.
• Figure 3: Schematic tree of the framework's hierarchical structure, with modules shown in square boxes. Care is taken to distinguish device-agnostic modules (responsible for collecting file names from directories) from device-specific ones, where derivative classes for different device species were implemented to account for differing file formats and extensions, number of channels of data, and different plotting/analysis requirements for each device.
• Figure 4: Schematic showing the code's image averaging pipeline, including how multiple background shots could be combined into a single "composite background". Multiple foreground shots could also be combined into a "composite foreground" shot. Subsequent analysis is performed on the output of the subtraction between the composite images.
• Figure 5: Sample output from one of the Chromox cameras, HRM3 (see Section \ref{['Section:FireballIII_experiment']}), tasked with monitoring the upstream cross-section of the plasma cell.