Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HPC resources for CMS offline computing: An integration and scalability challenge for the Submission Infrastructure

Antonio Perez-Calero Yzquierdo, Marco Mascheroni, Edita Kizinevic, Farrukh Aftab Khan, Hyunwoo Kim, Maria Acosta Flechas, Nikos Tsipinakis, Saqib Haleem

TL;DR

The paper addresses the challenge of integrating HPC resources into CMS offline computing through the CMS Submission Infrastructure (SI), which relies on federated HTCondor pools. It presents a systematic scalability evaluation of the SI, focusing on central manager throughput, slot negotiation, and scheduler capacity, complemented by real-world scale tests using GlideinWMS and HTCondor updates. The Spring 2023 results show that the Global Pool can reach around 800k simultaneous running jobs, revealing a fourfold growth potential relative to the then-current capacity, with collector saturation identified as a primary scaling limit. The findings underscore the viability of HPC integration for CMS, while highlighting the need for ongoing optimization and stress testing to meet HL-LHC-era compute demands.

Abstract

The computing resource needs of LHC experiments are expected to continue growing significantly during the Run 3 and into the HL-LHC era. The landscape of available resources will also evolve, as High Performance Computing (HPC) and Cloud resources will provide a comparable, or even dominant, fraction of the total compute capacity. The future years present a challenge for the experiments' resource provisioning models, both in terms of scalability and increasing complexity. The CMS Submission Infrastructure (SI) provisions computing resources for CMS workflows. This infrastructure is built on a set of federated HTCondor pools, currently aggregating 400k CPU cores distributed worldwide and supporting the simultaneous execution of over 200k computing tasks. Incorporating HPC resources into CMS computing represents firstly an integration challenge, as HPC centers are much more diverse compared to Grid sites. Secondly, evolving the present SI, dimensioned to harness the current CMS computing capacity, to reach the resource scales required for the HLLHC phase, while maintaining global flexibility and efficiency, will represent an additional challenge for the SI. To preventively address future potential scalability limits, the SI team regularly runs tests to explore the maximum reach of our infrastructure. In this note, the integration of HPC resources into CMS offline computing is summarized, the potential concerns for the SI derived from the increased scale of operations are described, and the most recent results of scalability test on the CMS SI are reported.

HPC resources for CMS offline computing: An integration and scalability challenge for the Submission Infrastructure

TL;DR

The paper addresses the challenge of integrating HPC resources into CMS offline computing through the CMS Submission Infrastructure (SI), which relies on federated HTCondor pools. It presents a systematic scalability evaluation of the SI, focusing on central manager throughput, slot negotiation, and scheduler capacity, complemented by real-world scale tests using GlideinWMS and HTCondor updates. The Spring 2023 results show that the Global Pool can reach around 800k simultaneous running jobs, revealing a fourfold growth potential relative to the then-current capacity, with collector saturation identified as a primary scaling limit. The findings underscore the viability of HPC integration for CMS, while highlighting the need for ongoing optimization and stress testing to meet HL-LHC-era compute demands.

Abstract

The computing resource needs of LHC experiments are expected to continue growing significantly during the Run 3 and into the HL-LHC era. The landscape of available resources will also evolve, as High Performance Computing (HPC) and Cloud resources will provide a comparable, or even dominant, fraction of the total compute capacity. The future years present a challenge for the experiments' resource provisioning models, both in terms of scalability and increasing complexity. The CMS Submission Infrastructure (SI) provisions computing resources for CMS workflows. This infrastructure is built on a set of federated HTCondor pools, currently aggregating 400k CPU cores distributed worldwide and supporting the simultaneous execution of over 200k computing tasks. Incorporating HPC resources into CMS computing represents firstly an integration challenge, as HPC centers are much more diverse compared to Grid sites. Secondly, evolving the present SI, dimensioned to harness the current CMS computing capacity, to reach the resource scales required for the HLLHC phase, while maintaining global flexibility and efficiency, will represent an additional challenge for the SI. To preventively address future potential scalability limits, the SI team regularly runs tests to explore the maximum reach of our infrastructure. In this note, the integration of HPC resources into CMS offline computing is summarized, the potential concerns for the SI derived from the increased scale of operations are described, and the most recent results of scalability test on the CMS SI are reported.
Paper Structure (4 sections, 2 figures)

This paper contains 4 sections, 2 figures.

Table of Contents

  1. Scalability limits in the CMS SI
  2. Pushing the limits of the CMS Global Pool
  3. The SI 2023 scale tests
  4. Conclusions and future work

Figures (2)

  • Figure 1: Number of CPU cores allocated at NERSC for CMS jobs execution as a function of time, demonstrating a pattern of resource provisioning in peaks usually found in relation to HPC facilities.
  • Figure 2: Final results of the Spring 2023 scalability tests of the CMS HTCondor Global Pool, reaching the simultaneous execution of approximately 800k jobs. The collector duty cycle indicates the fraction of seconds over a predefined time interval that the collector process was busy, i.e. not idle waiting to process incoming requests.