BORA: A Personalized Data Display for Large-scale Experiments

TL;DR

BORA addresses the need for reusable real-time data monitoring in large-scale physics experiments where petabyte-scale data rates rule out store-first analyses. It provides a browser-based, lightweight framework that encodes heterogeneous data as video streams and uses a server to parse diverse protocols, complemented by Jupyter notebooks for interactive scripting. The work demonstrates practical deployments (e.g., 22 KATRIN displays) and discusses integration with ADEI, Redis, and RTSP-based streams, with an evaluation showing WebRTC as the lowest-latency option. The system aims to simplify deployment, enable rapid customization, and support future upgrades and AI-assisted data exploration.

Abstract

Given the rapid improvement of the detectors at high-energy physics experiments, the need for real-time data monitoring systems has become imperative. The significance of these systems lies in their ability to display experiment status, steer software and hardware instrumentation, and provide alarms, thus enabling researchers to manage their experiments better. However, researchers typically build most data monitoring systems as standalone in-house solutions that cannot be reused for other experiments or future upgrades. We present BORA (personalized collaBORAtive data display), a lightweight browser-based monitoring system that supports diverse protocols and is built specifically for customizable visualization of complex data, which we standardize via video streaming. We show how absolute positioning layout and visual overlay background can address the diverse data display design requirements. Using the client-server architecture, we enable support for diverse communication protocols, with the server component responsible for parsing the incoming data. We integrate the Jupyter Notebook as part of our ecosystem to address the limitations of the web-based framework, providing a foundation to leverage scripting capabilities and integrate popular AI frameworks. Since video streaming is a core component of our framework, we evaluate viable approaches to streaming protocols like HLS, WebRTC, and MPEG-Websocket. The study explores the implications for our use case, highlighting its potential to transform data visualization and decision-making processes.

Figures (8)

• Figure 1: (a) A BORA status monitoring display for the Cryogenic Pumping Section segment of the KATRIN experiment and (b) the corresponding background image.
• Figure 2: Jupyter integration with BORA. (a) Define the data polling interval of BORA (2 seconds). (b) Add a list of sensors to the BORA widget with the identifier "container_1". (c) Generate a heatmap using a user-defined function, which pulls the data from a Redis database. Then, attach the output image to the BORA widget with the identifier "container_2". (d) Add a video URL to the BORA widget with the identifier "container_3".
• Figure 3: Comparison of the start-up delay and transmission latency between the HLS approach, the MPEG-Websocket approach, and the WebRTC approach.
• Figure 4: Data flow of the BORA framework.
• Figure 5: Components of the BORA architecture.