Introducing JIRIAF: A Virtual Kubelet Integration for Optimizing HPC Resource Provisioning

TL;DR

The paper presents JIRIAF, a framework for optimizing HPC resource provisioning across heterogeneous facilities by integrating Kubernetes with a Virtual Kubelet-based JIRIAF Resource Manager (JRM) that runs in userspace. It details the VK-Cmd implementation, pod lifecycle management, and HPA support, and demonstrates a proof-of-concept deployment on NERSC’s Perlmutter system for data-stream pipelines in CLAS12 ERSAP workloads. The work also introduces a Dynamic Bayesian Network-based digital twin to model and control a simulated queue for real-time monitoring, alongside Prometheus-based monitoring and FireWorks-driven JRM deployment. Together, these components illustrate a scalable, container-centric approach to distributed HPC resource management with practical deployment guidance and performance evaluation for real-world workloads.

Abstract

The JIRIAF (JLab Integrated Research Infrastructure Across Facilities) framework is designed to streamline resource management and optimize high-performance computing (HPC) workloads across heterogeneous environments. Central to JIRIAF is the JIRIAF Resource Manager (JRM), which effectively leverages Kubernetes and Virtual Kubelet to manage resources dynamically, even in environments with restricted user privileges. By operating in userspace, JRM facilitates the execution of user applications as containers across diverse computing sites, ensuring unified control and monitoring. The framework's effectiveness is demonstrated through a case study involving the deployment of data-stream processing pipelines on the Perlmutter system at NERSC, showcasing its capability to manage large-scale HPC applications efficiently. Additionally, we discuss the integration of a digital twin model for a simulated queue system related to a streaming system, using a Dynamic Bayesian Network (DBN) to enhance real-time monitoring and control, providing valuable insights into system performance and optimization strategies.

Figures (9)

• Figure 1: JIRIAF System Architecture and Workflow: This figure visually represents the sophisticated architecture of JIRIAF, emphasizing the roles and interconnectivity of its key components — the JFM, JCS, JRM, JMS, and JFE. By illustrating the flow of data and control across these components, the diagram elucidates the dynamic and efficient resource management system designed to optimize high-performance computing across heterogeneous environments.
• Figure 2: Flowchart illustrating the process of creating and monitoring the lifecycle of containers within a pod created by VK. The flowchart includes the initial creation of container states, updating pod status based on container states, and handling errors during the lifecycle. Key blocks indicate looping over containers, creating container states, updating pod status, and redirecting flows based on conditions.
• Figure 3: Network map illustrating the ports and SSH tunneling configurations used in the JRM deployment process. Arrowed red lines represent command execution, while black dashed lines indicate SSH tunneling initiated by JIRIAF2301. Solid gray lines represent listening connections, and dashed gray lines represent SSH tunneling from JRM at the compute node to login04 at NERSC. The map highlights communication paths for MongoDB, Kubernetes API server, JRM metrics, and custom metrics.
• Figure 4: Deployment and monitoring of Kubernetes applications with unique pod IPs using Prometheus Operator. The figure illustrates a Kubernetes application with two replicas, each assigned a unique IP address based on the DNS name of the node it runs on. Metrics are exported from each pod on ports 2221, 1776, and 8088, aggregated by a Service object, and monitored by a ServiceMonitor for Prometheus scraping.
• Figure 5: The ERSAP framework utilized in the JIRIAF deployment on the Perlmutter compute nodes at NERSC. The CLAS12 event reconstruction application ran on each node in the JIRIAF Kubernetes cluster, demonstrating the JIRIAF deployment's effectiveness in handling high-volume data-stream processing.