Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Advancing ATLAS DCS Data Analysis with a Modern Data Platform

Luca Canali, Andrea Formica, Michelle Ann Solis

TL;DR

The ATLAS DCS data are traditionally optimized for transactional storage, limiting large-scale analytics across time and devices. The authors introduce a modern data platform based on Apache Spark, the CERN Analytix Hadoop service, and Parquet storage, integrated with SWAN Jupyter notebooks to enable interactive, cross-dataset analysis, including DCS and conditions data. The NSW case study demonstrates rapid, scalable analytics for troubleshooting DAQ links and monitoring high voltage, illustrating practical benefits for detector operations. This framework provides a scalable, flexible, and accessible approach to data-driven detector monitoring with potential for cloud-native orchestration and machine learning in future work.

Abstract

This paper presents a modern and scalable framework for analyzing Detector Control System (DCS) data from the ATLAS experiment at CERN. The DCS data, stored in an Oracle database via the WinCC OA system, is optimized for transactional operations, posing challenges for large-scale analysis across extensive time periods and devices. To address these limitations, we developed a data pipeline using Apache Spark, CERN's Hadoop service, and the CERN SWAN platform. This framework integrates seamlessly with Python notebooks, providing an accessible and efficient environment for data analysis using industry-standard tools. The approach has proven effective in troubleshooting Data Acquisition (DAQ) links for the ATLAS New Small Wheel (NSW) detector, demonstrating the value of modern data platforms in enabling detector experts to quickly identify and resolve critical issues.

Advancing ATLAS DCS Data Analysis with a Modern Data Platform

TL;DR

The ATLAS DCS data are traditionally optimized for transactional storage, limiting large-scale analytics across time and devices. The authors introduce a modern data platform based on Apache Spark, the CERN Analytix Hadoop service, and Parquet storage, integrated with SWAN Jupyter notebooks to enable interactive, cross-dataset analysis, including DCS and conditions data. The NSW case study demonstrates rapid, scalable analytics for troubleshooting DAQ links and monitoring high voltage, illustrating practical benefits for detector operations. This framework provides a scalable, flexible, and accessible approach to data-driven detector monitoring with potential for cloud-native orchestration and machine learning in future work.

Abstract

This paper presents a modern and scalable framework for analyzing Detector Control System (DCS) data from the ATLAS experiment at CERN. The DCS data, stored in an Oracle database via the WinCC OA system, is optimized for transactional operations, posing challenges for large-scale analysis across extensive time periods and devices. To address these limitations, we developed a data pipeline using Apache Spark, CERN's Hadoop service, and the CERN SWAN platform. This framework integrates seamlessly with Python notebooks, providing an accessible and efficient environment for data analysis using industry-standard tools. The approach has proven effective in troubleshooting Data Acquisition (DAQ) links for the ATLAS New Small Wheel (NSW) detector, demonstrating the value of modern data platforms in enabling detector experts to quickly identify and resolve critical issues.
Paper Structure (10 sections, 4 figures)

This paper contains 10 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Data Storage, Pipeline and Analysis Framework
  3. Big Data Platform
  4. Analysis Framework
  5. Architecture Summary
  6. ATLAS NSW
  7. NSW Introduction
  8. Key Objectives Driving NSW DCS Data Analysis
  9. NSW DCS analysis with the Analytix platform
  10. Conclusions

Figures (4)

  • Figure 1: Overview of the Big Data architecture for Detector Control System (DCS) data analysis. The system integrates data from Oracle databases (including DCS, luminosity, and run information) and file-based metadata and mappings into the Hadoop ecosystem using Parquet files. Apache Spark serves as the core processing engine, enabling scalable analysis within an interactive environment powered by Jupyter notebooks on CERN SWAN.
  • Figure 2: Two examples of problematic MMG VTRx optical links over one year, querying DCS RSSI voltage data from over 500 links. Both links were periodically re-enabled throughout the year to check their status, since the nature of this VTRx failure is unpredictable. One link remains problematic with the RSSI staying high, while the other link is eventually re-enabled long term, showing a drop in RSSI.
  • Figure 3: MMG VTRx for all 16 sectors of one wheel which exhibited some unstable RSSI behavior. Each block represents one VTRx, with detector layer number on the x-axis and sector number on the y-axis. VTRx are selected on the z-axis if RSSI values are over 0.45 V and occurring more than three times, to filter out temporary behaviors, over the course of two years. Can also select links with a low RSSI standard deviation threshold to identify oscillating values.
  • Figure 4: Monitored high voltage for the 64 HV channels of one MMG sector. The number of channels with a daily maximum voltage above and below the nominal voltage of 505 V, during data-taking in 2024. Periods where channels are below nominal correspond to special runs where NSW high voltage is off. DCS data joined with Tier 0 processed runs to select times during running in ATLAS.